Front incident
Increased error rate sending and receiving messages on email-based channels
Front experienced a major incident on September 27, 2024 affecting SMTP (non-Gmail / O365) and Gmail and 1 more component, lasting 6h 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 27, 2024, 08:25 PM UTC
We are currently investigating this issue.
- investigating Sep 27, 2024, 08:25 PM UTC
We are continuing to investigate this issue.
- identified Sep 27, 2024, 08:38 PM UTC
We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.
- identified Sep 27, 2024, 09:02 PM UTC
We are validating the fix in our staging environment.
- identified Sep 27, 2024, 09:27 PM UTC
We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.
- monitoring Sep 27, 2024, 09:47 PM UTC
The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.
- monitoring Sep 28, 2024, 02:21 AM UTC
Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.
- resolved Sep 28, 2024, 02:51 AM UTC
This incident has been resolved.