LogDNA experienced a major incident on October 28, 2021 affecting Alerting, lasting 5h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 28, 2021, 06:13 PM UTC
Our email alerting feature is not working at the moment and customers are not receiving alerts by email in US region. This is due to an ongoing incident with our email provider; see https://status.sparkpost.com/incidents/bwl8dr6gwmts?u=ydzrh5x205pf for more detail. Other types of alerts, such as Slack and webhook, are still working. We are investigating.
- identified Oct 28, 2021, 09:02 PM UTC
Our email provider reports that outbound message delivery has resumed but it is not yet fully operational. Our provider will keep unsent emails in their queue and continue to try to send them.
- resolved Oct 28, 2021, 11:22 PM UTC
Our email alerting feature has been restored to normal operation. All services are fully functional.
- postmortem Nov 03, 2021, 06:15 PM UTC
**Start Time:** Thursday, October 28, 2021, at 16:56:52 UTC **End Time:** Thursday, October 28, 2021, at 22:17:24 UTC **Duration:** 5:20:32 **What happened:** Email notifications of all kinds, including from alerts, were delayed for about 5 hours. Notifications sent by Slack and Webhooks were not affected. **Why it happened:** Our email service provider \(Sparkpost\) experienced an incident that caused delays for all emails from the LogDNA service. We rely on this service to deliver email of all kinds, including notifications for alerts. Email messages were delayed and queued until our email service provider was able to recover. More information on the incident can be found at Sparkpost’s Status Page: [https://status.sparkpost.com/incidents/bwl8dr6gwmts?u=ydzrh5x205pf](https://status.sparkpost.com/incidents/bwl8dr6gwmts?u=ydzrh5x205pf) **How we fixed it:** No remedial action was possible by LogDNA. We waited until the incident from Sparkpost, our email hosting provider, was resolved. **What we are doing to prevent it from happening again:** For this type of incident, LogDNA cannot take proactive preventive measures.