Sleuth incident

We are experiencing downtime related to a bad migration

Critical Resolved View vendor source →

Sleuth experienced a critical incident on February 11, 2021 affecting website and impact tracking, lasting 1h 41m. The incident has been resolved; the full update timeline is below.

Started
Feb 11, 2021, 04:51 PM UTC
Resolved
Feb 11, 2021, 06:32 PM UTC
Duration
1h 41m
Detected by Pingoru
Feb 11, 2021, 04:51 PM UTC

Affected components

websiteimpact tracking

Update timeline

  1. investigating Feb 11, 2021, 04:51 PM UTC

    We're investigating the cause now

  2. investigating Feb 11, 2021, 04:54 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Feb 11, 2021, 05:36 PM UTC

    We have cleared the problem. A migration locked our main deploys table and killing the initiating process did not clear the lock. We've cleared the lock and the site has resumed it's normal functioning. We're monitoring to make sure everything is completely back to normal.

  4. resolved Feb 11, 2021, 06:32 PM UTC

    We've successfully re-run an updated version of the migration and all systems are back to normal.