Front incident
Delays in sending and receiving Gmail messages [gmail]
Front experienced a major incident on September 13, 2024 affecting Gmail, lasting 3h 47m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 13, 2024, 03:38 PM UTC
We're observing problems sending and receiving some messages on Gmail channels in the us-west-2 region. We're working on identifying the root cause.
- investigating Sep 13, 2024, 03:52 PM UTC
We are continuing to investigate this issue, which seems to be related to Gmail's rate limiting mechanisms. We're seeing some impact across multiple regions.
- investigating Sep 13, 2024, 04:35 PM UTC
We are continuing to see message delays, and are rescaling our internal queues globally to help mitigate the issue.
- investigating Sep 13, 2024, 04:57 PM UTC
We are beginning to see a gradual increase in successful deliveries. We are continuing to monitor the situation, as well as replaying deliveries that have been delayed by the original incident.
- identified Sep 13, 2024, 05:22 PM UTC
We are on the recovery path now and are focused on draining the backlog of delayed messages.
- monitoring Sep 13, 2024, 05:44 PM UTC
We are continuing to drain the backlog and will be doing so for the next couple hours.
- monitoring Sep 13, 2024, 06:38 PM UTC
Continuing to drain the backlog
- monitoring Sep 13, 2024, 07:19 PM UTC
Some channels are still experiencing delays as we import the backlog of messages received while Gmail was experiencing an outage
- resolved Sep 13, 2024, 07:25 PM UTC
All channels should be back to normal now.