Dead Man's Snitch incident

Check-In Processing Delays

Minor Resolved View vendor source →

Dead Man's Snitch experienced a minor incident on November 25, 2016, lasting 1h 37m. The incident has been resolved; the full update timeline is below.

Started
Nov 25, 2016, 04:01 PM UTC
Resolved
Nov 25, 2016, 05:39 PM UTC
Duration
1h 37m
Detected by Pingoru
Nov 25, 2016, 04:01 PM UTC

Update timeline

  1. identified Nov 25, 2016, 04:01 PM UTC

    Our primary queue server is currently having CPU issues causing check-ins to take longer to both process and respond to clients. We're working with our service provider to address the issue. We've failed over check-in processing to a secondary service but are having issues keeping up with message rates. We expect check-ins processing to be delayed until our main queue server is back at full capacity.

  2. monitoring Nov 25, 2016, 04:14 PM UTC

    The issues with our primary queue server appear to be fixed and we've reenabled it for check-in processing. We are handling all new check-ins as they arrive but are still working off the backlog of check-ins from our backup queue service.

  3. monitoring Nov 25, 2016, 04:26 PM UTC

    We've finished processing all check-ins that were left on our failover queue. Everything is looking to be back to normal. We'll continue to monitor the situation in case of further issues.

  4. resolved Nov 25, 2016, 05:39 PM UTC

    Everything is looking good and has been stable the last hour 🎉