Dead Man's Snitch incident

False Alerts for short intervals Snitches

Minor Resolved View vendor source →

Dead Man's Snitch experienced a minor incident on April 8, 2025 affecting Snitch Check-in Processing, lasting 3h 14m. The incident has been resolved; the full update timeline is below.

Started
Apr 08, 2025, 12:12 AM UTC
Resolved
Apr 08, 2025, 03:26 AM UTC
Duration
3h 14m
Detected by Pingoru
Apr 08, 2025, 12:12 AM UTC

Affected components

Snitch Check-in Processing

Update timeline

  1. identified Apr 08, 2025, 12:12 AM UTC

    We have received reports of false alerts being sent out for some short interval Snitches. The issue is related to maintenance an upstream provider is performing on our main message bus service that is causing some check-ins to be processed out of order. We've taken an initial step to mitigate the issue but have more work to do to fully address the issue.

  2. monitoring Apr 08, 2025, 01:02 AM UTC

    We are continuing to monitor the situation as maintenance on our primary message bus continues at our upstream provider. We're currently discussing options for a longer term fix but we expect that fully solving the issue will take longer than it will take for the maintenance to complete.

  3. resolved Apr 08, 2025, 03:26 AM UTC

    Our primary message bus is still undergoing maintenance and we have implemented all the mitigations that we can without making larger changes. Taking that into consideration and the fact that we have not seen a disruption which would cause out of order processing within the last hour we've decided to close this incident.