Dead Man's Snitch experienced a minor incident on January 28, 2025 affecting iOS Push Notification Alerts and Email Alerts, lasting 59m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 28, 2025, 02:39 PM UTC
The system we use to send alerts and emails is currently backlogged due an upstream service issue. We're
- identified Jan 28, 2025, 02:48 PM UTC
We believe we've identified the issue which has been causing one job (sending iOS push notifications) to block our queue. We're working to deploy a fix now.
- monitoring Jan 28, 2025, 02:57 PM UTC
We've deployed a fix for the issue and our worker system has worked through all of the queued jobs. We're monitoring to make sure the system is stable and will be reviewing steps to avoid this in the future.
- resolved Jan 28, 2025, 03:39 PM UTC
Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.