MailChannels experienced a minor incident on April 22, 2024 affecting SMTP Service, lasting 13h 54m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 22, 2024, 06:44 AM UTC
Severity Level: 3 We are currently investigating higher than normal connection latency that might have an impact on email delivery services.
- investigating Apr 22, 2024, 09:29 AM UTC
We are continuing to investigate the issue.
- investigating Apr 22, 2024, 12:25 PM UTC
We are continuing to investigate the issue.
- investigating Apr 22, 2024, 12:39 PM UTC
Message delivery is slow for a small percentage of traffic. We are trying to clear out all the messages in our queues as fast as possible.
- investigating Apr 22, 2024, 03:16 PM UTC
Legitimate emails backed up in our queues have been delivered. We are still seeing increased latency at the ingress side. We are continuing to investigate.
- investigating Apr 22, 2024, 06:04 PM UTC
The root cause has been identified as a failure of certain auto-scaling functionality within the service that failed to adapt service scale to a surge of incoming email. We will provide a more detailed analysis when this incident is fully resolved.
- monitoring Apr 22, 2024, 06:34 PM UTC
All of our metrics, mainly SMTP connection delay and SMTP round trip delivery time, have returned to normal as of 18:00 UTC. This means that if there are any emails backed up in the server queues, they should start delivering. Customers may continue to receive complaints from users about delivery delays because messages may still be leaving the queue as per the exponential back-off rules of the SMTP servers. However, new messages introduced into the service will be delivered immediately. One option is to flush the queues. If you are still encountering any issues, please reach out to [email protected]
- resolved Apr 22, 2024, 08:38 PM UTC
All of the metrics, mainly SMTP connection delay and SMTP round trip delivery time, continue to be normal. If you are still encountering any issues, please reach out to [email protected]. We will be updating the port-mortem here shortly.
- postmortem Apr 29, 2024, 02:12 PM UTC
The postmortem is posted here - [https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/](https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/)