Dead Man's Snitch incident

Delayed Alerts

Notice Resolved View vendor source →

Dead Man's Snitch experienced a notice incident on March 27, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Mar 27, 2023, 07:21 PM UTC
Resolved
Mar 27, 2023, 05:30 PM UTC
Duration
Detected by Pingoru
Mar 27, 2023, 07:21 PM UTC

Update timeline

  1. resolved Mar 27, 2023, 07:21 PM UTC

    Starting at 17:15 UTC until 18:45 UTC we had an issues where alerts for missing and errored Snitches were delayed. All history for Snitches is accurate despite the delay. We've identified the root cause as a bad deploy which caused issues connecting to our backend message bus. We were not initially alerted to the issue and we're investigating the gap in our alerting to make sure this doesn't happen again.