Courier had 25 outages in the last 2 years totaling 60h 37m of downtime — averaging 1 incident per month.
There were 25 Courier outages since October 10, 2024 totaling 60h 37m of downtime. Each is summarised below — incident details, duration, and resolution information.
Messages are being queued resulting in delayed sends.
We found the issue. We believe it's an issue with our cloud provider. Fix is being rolled out.
Messages are flowing again. We will continue to monitor.
All queued messages have been processed.
We are currently investigating an issue with delayed messages.
Issue has been identified and are working to roll out a fix. All messages that have been queued up will be sent.
The fix as been deployed and the queued messages are beginning to be processed again
All queued messages have been processed. We will continue to monitor.
This incident has been resolved.
The team has identified a regression causing send requests to be stuck in a queued state. We have initiated a rollback to quickly resolve the issue. More updates to follow
The rollback deploy will land in about 30 minutes.
The team has identified that the most recent deployment was not the cause of the regression. The team is still working on identifying the root cause. Messages will not be dropped, and they will be processed once the fix is out.
Messages are being processed while our pipeline continues to process enqueued messages.
Send pipeline has resolved and enqueued messages are now sending properly.
Slack has reported messaging, and app degredation on their platform that could be impacting deliverability of Slack messages. This is a downstream provider issue and can be monitored on Slack's statuspage: https://slack-status.com/
Please continue to monitor Slack's statuspage for any updates related to message deliverability: https://slack-status.com/
The Courier team identified an issue affecting observability metrics between 21:38 UTC and 22:13 UTC, during which metrics briefly experienced an outage. A fix was released and metrics are stabilized through observability channels. Metrics received during the outage window will not be accounted for in observability dashboards.
The Courier team was alerted of a small degredation in the Courier web application. We have identified the issues and have rolled out a fix to mitigate. The current features will encounter some latency: - Audience Calculation - Audit Trail logging We will follow up with updates.
Fix is out and application is stable
An issue with our service provider, AWS, has impacted all Courier services. The engineering team is working on resolution.
We are continuing to work on a fix for this issue.
Service has been restored. Engineering is working through system review and will continue to monitor.
The incident has been resolved
The Courier team was alerted of an issue with MS Teams requests experiencing errors sending. We have identified the issue and published a fix that will land in production shortly
Release is in production and Microsoft Teams messages are passing successfully.
Courier experienced a slowdown in message delivery with Mailgun providers around 2AM PST. Our monitoring systems caught the spike in the bottleneck, which eventually stabilized. Our infrastructure team is actively monitoring.
Courier's delivery platform experienced significant latency around 1am PST. Our platform team managed to resolve the issue and is actively monitoring for any potential issues.
An issue with logging in to the Courier platform and overall app performance was identified by our team and a revert of the offending release has been deployed. The estimated time for the release to land is 45 minutes.
Revert has landed and webapp is operational.
Cloudflare has reported an outage that will impact SSO features provided by WorkOS. Customers with Okta SSO might experience some degradation when signing in. Our team is monitoring the status for both Cloudflare and WorkOS. WorkOS status: https://status.workos.com/incidents/k9s870cktcsf Cloudflare Status: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
WorkOS systems are back to normal.
The Courier team has identified an issue where there are significant delays of up to 10 minutes for messages. We are currently investigating and looking for a root cause to mitigate. Updates will follow.
Backed-up messages were caused by a bottleneck in our system, which has started to catch up. Messages will stabilize as the bottleneck clears.
Message bottleneck has cleared and the queue has stabilized. We will continue to monitor.
Messages have stabilized but we will continue to monitor.
Messages have recovered and pipeline is stable again.
We are currently investigating an issue where EU workspace notifications appear to remain in the “queued” state.
The issue has been identified and a fix is being implemented.
The issue has been resolved. The root cause was purely a logging issue, notifications were still sent but showing up in logs as queued. All logs have been rehydrated
The Courier team has identified an issue with a recent deployment that is impacting AWS SES integrations. We have found the root cause and have reverted the changes. The team is gathering impact.
The reverted changes will land in production in around one hour. The team will investigate the impact and handle message retries accordingly.
Fix has landed and the team is monitoring progress. Messages sent to AWS SES are sending normally. The team will reprocess impacted messages.
The team is still working through the reprocessing for impacted messages. Confirmation of all impacted messages reprocessed will be updated here.
All impacted messages have been reprocessed.
The Courier team has identified an issue where messages fail due to an "Internal Courier Error" in the test environment. Our team found the root cause and are working to mitigate the issue.
We have tested the fix for "Internal Courier Error" and will be releasing it to production soon.
A fix has been merged and will fully deploy within the next hour. Updates to follow.
The fix has landed in production and we will be monitoring for any more "Internal Courier Errors". We will be working on reprocessing any dropped messages caused by this issue.
Messages with tags are no longer impacted, and error has been resolved. The team is working on reprocessing dropped messages.
The team is still working on reprocessing dropped messages caused by this issue. Send pipeline is operational.
All impacted messages have been reprocessed.
The Courier team has identified a slowdown in Segment event processing. There will be a slight delay for messages triggered by incoming Segment triggers. We estimate the queue to normalize soon and are monitoring.
Segment event processing stream has resolved.
The Courier team identified a delay in message processing and event updating. The pipeline queue for message delivery has normalized, however significant slowdowns in message events are still processing.
We're monitoring our processing pipeline as messages are flowing normally. There is a slight delay in event processing which will also impact outbound webhooks.
Message sends are flowing normally, and events are caught up without delays.
The Courier team has identified a slowdown in sending notifications. This issue has been identified by our team and is closely monitoring. We do not expect any messages to drop. We will continue to monitor and update accordingly.
Our team has identified a bottleneck in our system caused by timeouts on our sendworker. Our exponential backoff has caught up and messages are starting to go through.
The team has identified the issue related to network connections on AWS which resulted in a backup of messages on our pipeline. Messages are flowing through as expected, and the stream has caught up.
The Courier team has identified a slowdown in our pipeline. The queue has backed up but no messages have been dropped at the moment. We will continue to update as we monitor our pipeline.
The team has identified the cause of message sends and slowdowns to be related to an AWS service. Messages that were stuck in a queued state are slowly passing through. The team will continue to monitor and update.
The send stream has caught up, all messages have been sent and the issue appears to be fully resolved.
We are currently experiencing an issue affecting automation workflows that include delay steps. Some customers may encounter failures when attempting to execute automations with scheduled delays. Our engineering team has identified the root cause and is implementing a fix. We expect the service to be fully restored in a short while.
We have deployed a fix for the earlier issue affecting automation workflows with delay steps. New automation workflows are now executing successfully. However, automations that failed during the incident period (approximately 3:20 PM PT - 6:05 PM PT) have not yet been automatically retried. Our engineering team is monitoring the recovery and evaluating a plan to process these backlogged automations.
The service issue affecting automation workflows with delay steps has been partially resolved. New automation workflows started after 6:05 PM PT are executing successfully. However, we have determined that automations that failed during the incident period (approximately 3:20 PM PT - 6:05 PM PT) cannot be automatically retried at this time due to technical limitations. If your business was impacted by failed automation runs during this incident, please contact our support team and we will work directly with you to address your concerns. We sincerely apologize for this disruption to your workflows.
Courier has identified a delay in message processing for the messages API. We are currently investigating the root cause and will update you periodically.
We are continuing to monitor the root cause of the delayed message processing.
The team is still investigating and discussing the root cause internally. Messages are experiencing longer than normal processing times.
The team has identified the issue and is rolling out a fix for the message delays.
Fix has been deployed and will be live in around 1hr.
The team identified an issue with the latest fix and reverted it. A new fix has been published to mitigate the message delay issue. ETA to land in production ~45 minutes.
A fix has been implemented and delivery times are beginning to returning to expected levels. The team will continue to monitor.
This incident has been resolved
The Courier team is investigating an issue with the event logger for message event logs hitting a bottleneck. The team is actively investigating. Messages are still sending.
We are continuing to investigate this issue.
The team is testing out a fix to reduce the bottlenecked log lines before releasing to production.
The team encountered an issue with testing the fix and reverted the update. We are publishing a new update that should resolve the backlogged message logs.
The fix has landed in production and the team is monitoring the message log queue. Message event logs should be flowing normally.
The data stream is unblocked, and the message logs queue is resolved and flowing normally.
The Courier Team is investigating an issue related to Inbox fetching messages in the component.
The team has identified the issue and will be rolling out a fix. The team will reprocess the impacted messages that were not fetched.
Inbox connectivity for fetching messages has been reestablished. The team is monitoring closely, and working on retroactively processing impacted messages to Inbox
We are continuing to gather a list of impacted messages to backfill impacted Inbox components.
We are continuing to monitor for any further issues.
The team has gathered impacted messages and is running tests before a release.
The team has initiated backlogging the impacted Inbox messages.
Inbox connectivity issues resolved.