Baremetrics incident

Networking issues

Major Resolved View vendor source →

Baremetrics experienced a major incident on February 21, 2018, lasting 6h 4m. The incident has been resolved; the full update timeline is below.

Started
Feb 21, 2018, 02:47 PM UTC
Resolved
Feb 21, 2018, 08:51 PM UTC
Duration
6h 4m
Detected by Pingoru
Feb 21, 2018, 02:47 PM UTC

Update timeline

  1. investigating Feb 21, 2018, 02:47 PM UTC

    We're seeing issues across multiple services, which appears to be a wider networking issue with our ISP. We've paused metric updates while we investigate. You may encounter the occasional internal server error. We'll update as we know more.

  2. investigating Feb 21, 2018, 03:03 PM UTC

    We've paused Recover and Report emails as we continue to investigate. These will queued up and sent once the networking issue is resolved.

  3. investigating Feb 21, 2018, 04:37 PM UTC

    We're putting the dashboard in Maintenance Mode while we work with network providers to resolve the issue.

  4. investigating Feb 21, 2018, 05:53 PM UTC

    Still working through network issues. We're currently actively debugging issues with the network from multiple points within our infrastructure to narrow down the problem. Still appears to be a higher level network issue with our ISP outside of our direct control, but we're exhausting all options.

  5. monitoring Feb 21, 2018, 06:00 PM UTC

    We believe the networking issue has been resolved, though we're actively monitoring and working with network providers to verify and be sure everything has stabilized.

  6. monitoring Feb 21, 2018, 07:19 PM UTC

    Still monitoring network activity to make sure everything has stabilized. We've resumed metric processing along with Recover and report emails. Most accounts are caught back up now.

  7. resolved Feb 21, 2018, 08:51 PM UTC

    We believe the issue is resolved, though we'll continue to monitor to be sure.