Resolved after 59 minutes, affecting iOS Push Notification Alerts and Email Alerts, on January 28, 2025.
Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.
Affected components
Update timeline
- identified Jan 28, 2025, 02:39 PM UTC
The system we use to send alerts and emails is currently backlogged due an upstream service issue. We're
- identified Jan 28, 2025, 02:48 PM UTC
We believe we've identified the issue which has been causing one job (sending iOS push notifications) to block our queue. We're working to deploy a fix now.
- monitoring Jan 28, 2025, 02:57 PM UTC
We've deployed a fix for the issue and our worker system has worked through all of the queued jobs. We're monitoring to make sure the system is stable and will be reviewing steps to avoid this in the future.
- resolved Jan 28, 2025, 03:39 PM UTC
Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.
Your product runs on more than Dead Man's Snitch
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services