Dead Man's Snitch incident

Delayed Check-In Processing

Notice Resolved View vendor source →

Dead Man's Snitch experienced a notice incident on May 4, 2018, lasting 3h 13m. The incident has been resolved; the full update timeline is below.

Started
May 04, 2018, 10:57 AM UTC
Resolved
May 04, 2018, 02:11 PM UTC
Duration
3h 13m
Detected by Pingoru
May 04, 2018, 10:57 AM UTC

Update timeline

  1. investigating May 04, 2018, 10:57 AM UTC

    We have sent out several false alerts due to delays in check-in processing. We're still investigating the root cause but have mitigated the issue for now.

  2. monitoring May 04, 2018, 11:30 AM UTC

    As of 10:44 UTC we've caught up on the back log of check-ins and the system is back to being stable. We're still investigating the root cause and will be preparing a post mortem once we know more. At this point we're keeping a close eye on the system to make sure this doesn't impact anyone should it start happening again.

  3. resolved May 04, 2018, 02:11 PM UTC

    System has been stable for several hours. We have a plan in place to automate future occurances that we will be deploying soon.