Courier experienced a minor incident on October 21, 2024 affecting Observability, lasting 2h 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 21, 2024, 03:56 PM UTC
The Courier team is investigating an issue with the event logger for message event logs hitting a bottleneck. The team is actively investigating. Messages are still sending.
- investigating Oct 21, 2024, 03:56 PM UTC
We are continuing to investigate this issue.
- identified Oct 21, 2024, 04:07 PM UTC
The team is testing out a fix to reduce the bottlenecked log lines before releasing to production.
- identified Oct 21, 2024, 05:19 PM UTC
The team encountered an issue with testing the fix and reverted the update. We are publishing a new update that should resolve the backlogged message logs.
- monitoring Oct 21, 2024, 06:30 PM UTC
The fix has landed in production and the team is monitoring the message log queue. Message event logs should be flowing normally.
- resolved Oct 21, 2024, 06:51 PM UTC
The data stream is unblocked, and the message logs queue is resolved and flowing normally.