Sleuth incident

We're seeing site-wide slowdowns, we're investigating an increase in DB operations

Minor Resolved View vendor source →

Sleuth experienced a minor incident on November 30, 2021 affecting website and impact tracking, lasting 1h 14m. The incident has been resolved; the full update timeline is below.

Started
Nov 30, 2021, 10:25 PM UTC
Resolved
Nov 30, 2021, 11:39 PM UTC
Duration
1h 14m
Detected by Pingoru
Nov 30, 2021, 10:25 PM UTC

Affected components

websiteimpact tracking

Update timeline

  1. investigating Nov 30, 2021, 10:25 PM UTC

    We are seeing slow downs related to increased queries against our database. We are investigating the cause.

  2. monitoring Nov 30, 2021, 11:08 PM UTC

    We've identified the problem and remediated it. We had a few very long running queries get stuck in our database which had negative follow on effects. We've killed the offending queries and removed the code that trigged them. We'll be following up with a change to stop this kind of vector in the near future.

  3. resolved Nov 30, 2021, 11:39 PM UTC

    This incident has been resolved.