Dead Man's Snitch experienced a notice incident on December 15, 2021, lasting 1h 21m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Dec 15, 2021, 11:53 PM UTC
We're investigating an issue related to a backend migration that caused some previously paused Snitches to become unpaused and start alerting again. We're working to repause those Snitches.
- identified Dec 16, 2021, 12:20 AM UTC
We've identified that some Snitches that were pending (i.e. have never checked-in to Dead Man's Snitch) were alerting. We've reverted those Snitches which were pending and hadn't been paused to their original state. This should stop future emails for those Snitches.
- monitoring Dec 16, 2021, 12:39 AM UTC
Pending Snitches should all be fixed and we're seeing alert metrics return to normal. We're waiting until the top of the hour after our basic hourly Snitches alert before we consider the incident closed.
- resolved Dec 16, 2021, 01:15 AM UTC
We're really sorry for the trouble and confusion this caused. The issue itself was related to migrating to a new alerting backend that's been in the works for quite a while. We had hoped for it to be seamless (and it almost was) but we failed to double check the state of Snitches that were waiting for their first check-in. We plan to take another pass at our verification process to make sure we haven't missed anything else.