Courier experienced a major incident on August 19, 2024 affecting Web Application and API and 1 more component, lasting 3h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Web ApplicationAPIObservability
Update timeline
- identified Aug 19, 2024, 06:32 PM UTC
The Courier team identified an issue in our health monitoring involving our message event processing. The issue has been identified and a revert is in place.
- monitoring Aug 19, 2024, 06:48 PM UTC
Our team has released a revert to address the regression and it's in the process of merging.
- monitoring Aug 19, 2024, 06:50 PM UTC
The release is live, and the team is monitoring it.
- monitoring Aug 19, 2024, 06:56 PM UTC
Release is published and building to production. ETA ~45 minutes.
- monitoring Aug 19, 2024, 08:02 PM UTC
Fix has been deployed, and enqueued messages have started to go through slowly. Once the bottleneck clears, messages should start to flow normally.
- resolved Aug 19, 2024, 09:35 PM UTC
The general pipeline has recovered.