Sleuth incident

We are seeing some issues with our Redis instance

Minor Resolved View vendor source →

Sleuth experienced a minor incident on October 15, 2021 affecting website and impact tracking, lasting 1h 2m. The incident has been resolved; the full update timeline is below.

Started
Oct 15, 2021, 12:25 PM UTC
Resolved
Oct 15, 2021, 01:27 PM UTC
Duration
1h 2m
Detected by Pingoru
Oct 15, 2021, 12:25 PM UTC

Affected components

websiteimpact tracking

Update timeline

  1. investigating Oct 15, 2021, 12:25 PM UTC

    Users may see some sporadic errors and impact processing may be delayed. We are investigating the issue.

  2. monitoring Oct 15, 2021, 01:06 PM UTC

    We identified the issue. Our background tasks were creating keys that weren't being cleaned up and eventually chewed up most of our storage. We've cleared those keys and are putting in place a way to stop this from happing moving forward. The service is now restored to normal and we are monitoring.

  3. resolved Oct 15, 2021, 01:27 PM UTC

    We're back to fully operational.