Courier experienced a minor incident on February 28, 2025 affecting API, lasting 11h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 28, 2025, 04:48 PM UTC
The Courier team has identified an issue where messages fail due to an "Internal Courier Error" in the test environment. Our team found the root cause and are working to mitigate the issue.
- identified Feb 28, 2025, 05:14 PM UTC
We have tested the fix for "Internal Courier Error" and will be releasing it to production soon.
- identified Feb 28, 2025, 05:36 PM UTC
A fix has been merged and will fully deploy within the next hour. Updates to follow.
- monitoring Feb 28, 2025, 06:40 PM UTC
The fix has landed in production and we will be monitoring for any more "Internal Courier Errors". We will be working on reprocessing any dropped messages caused by this issue.
- monitoring Feb 28, 2025, 06:57 PM UTC
Messages with tags are no longer impacted, and error has been resolved. The team is working on reprocessing dropped messages.
- monitoring Feb 28, 2025, 07:52 PM UTC
The team is still working on reprocessing dropped messages caused by this issue. Send pipeline is operational.
- resolved Mar 01, 2025, 04:11 AM UTC
All impacted messages have been reprocessed.