Zipline incident

Investigating elevated Error Rates

Minor Resolved View vendor source →

Zipline experienced a minor incident on October 5, 2020 affecting Core Platform, lasting 48m. The incident has been resolved; the full update timeline is below.

Started
Oct 05, 2020, 04:21 PM UTC
Resolved
Oct 05, 2020, 05:09 PM UTC
Duration
48m
Detected by Pingoru
Oct 05, 2020, 04:21 PM UTC

Affected components

Core Platform

Update timeline

  1. investigating Oct 05, 2020, 04:21 PM UTC

    Our monitors are reporting elevated error rates affecting a portion of user traffic. We're investigating the cause and will report as soon as we have more.

  2. monitoring Oct 05, 2020, 04:34 PM UTC

    The elevated error rates have subsided. We've continuing to monitor the site and investigating the cause.

  3. resolved Oct 05, 2020, 05:09 PM UTC

    Everything has been resolved and error rates have subsided as of 9:14 PST. In total, it was 5 minutes of increased error messages impacting a portion of our users. Most users saw no issues. The error rates were due to an overload of the system because of a runaway database backup. We're will be revising our backup strategy to ensure this does not happen again.