Courier Outage History
Courier is up right nowPingoru has recorded 52 incidents for Courier: 7 in the last 12 months and 2 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 0 | 0 of 720 (0%) |
| May 2026 | 1 | 0 of 744 (0%) |
| June 2026 | 1 | 2 of 720 (0%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 2 | 1 of 519 (0%) |
Incidents, newest first
Minor October 10, 2026 Queued Messages 14m API
Started Oct 10, 2026, 12:25 AM UTC· resolved Oct 10, 2026, 12:40 AM UTC
- investigating Oct 10, 2026, 12:25 AM UTC
Messages are being queued resulting in delayed sends.
- identified Oct 10, 2026, 12:31 AM UTC
We found the issue. We believe it's an issue with our cloud provider. Fix is being rolled out.
- monitoring Oct 10, 2026, 12:32 AM UTC
Messages are flowing again. We will continue to monitor.
- resolved Oct 10, 2026, 12:40 AM UTC
All queued messages have been processed.
Minor October 9, 2026 Delayed Messages 36m API
Started Oct 09, 2026, 11:15 PM UTC· resolved Oct 09, 2026, 11:52 PM UTC
- investigating Oct 09, 2026, 11:15 PM UTC
We are currently investigating an issue with delayed messages.
- identified Oct 09, 2026, 11:26 PM UTC
Issue has been identified and are working to roll out a fix. All messages that have been queued up will be sent.
- monitoring Oct 09, 2026, 11:39 PM UTC
The fix as been deployed and the queued messages are beginning to be processed again
- monitoring Oct 09, 2026, 11:42 PM UTC
All queued messages have been processed. We will continue to monitor.
- resolved Oct 09, 2026, 11:52 PM UTC
This incident has been resolved.
Minor June 18, 2026 Send Pipeline Latency 1h 57m API
Started Jun 18, 2026, 08:11 PM UTC· resolved Jun 18, 2026, 10:09 PM UTC
- identified Jun 18, 2026, 08:11 PM UTC
The team has identified a regression causing send requests to be stuck in a queued state. We have initiated a rollback to quickly resolve the issue. More updates to follow
- identified Jun 18, 2026, 08:35 PM UTC
The rollback deploy will land in about 30 minutes.
- identified Jun 18, 2026, 09:31 PM UTC
The team has identified that the most recent deployment was not the cause of the regression. The team is still working on identifying the root cause. Messages will not be dropped, and they will be processed once the fix is out.
- monitoring Jun 18, 2026, 09:53 PM UTC
Messages are being processed while our pipeline continues to process enqueued messages.
- resolved Jun 18, 2026, 10:09 PM UTC
Send pipeline has resolved and enqueued messages are now sending properly.
Minor May 27, 2026 Slack Deliverability Issues 10m
Started May 27, 2026, 11:06 PM UTC· resolved May 27, 2026, 11:16 PM UTC
- monitoring May 27, 2026, 11:06 PM UTC
Slack has reported messaging, and app degredation on their platform that could be impacting deliverability of Slack messages. This is a downstream provider issue and can be monitored on Slack's statuspage: https://slack-status.com/
- resolved May 27, 2026, 11:16 PM UTC
Please continue to monitor Slack's statuspage for any updates related to message deliverability: https://slack-status.com/
Critical February 2, 2026 Courier Multi Service Outage 3h 6m Web Application, API, Automations, Courier Inbox
Started Feb 02, 2026, 08:07 PM UTC· resolved Feb 02, 2026, 11:13 PM UTC
- identified Feb 02, 2026, 08:07 PM UTC
The Courier team has encountered an issue on our platform impacting several services. The team has rolled out a redeploy to roll back changes.
- identified Feb 02, 2026, 08:17 PM UTC
Due to GH Actions experiencing an outage(https://www.githubstatus.com/), the redeploy took longer than expected and are now cutting a release.
- identified Feb 02, 2026, 08:29 PM UTC
We are continuing to work on a fix for this issue. The following services are still impacted: - Login - Webapp
- identified Feb 02, 2026, 08:44 PM UTC
The current outage still impacts the current services: - webhooks - login - API endpoints - FE access. Our team is actively working on the redeployment.
- identified Feb 02, 2026, 09:03 PM UTC
The Courier team has a redeploy ready and waiting on GH Actions to recover from their degradation to publish the release.
- identified Feb 02, 2026, 09:23 PM UTC
The team is exploring alternate methods to redeploy the release until GH Actions recovers.
- identified Feb 02, 2026, 09:50 PM UTC
The team has rolled back the deployment manually bypassing GH Actions outage. We're waiting for the deployment to land and confirm that all core functionality has been restored.
- monitoring Feb 02, 2026, 10:01 PM UTC
The manual rollback has successfully landed, and all services are operational. The team is diligently monitoring and testing other services.
- monitoring Feb 02, 2026, 10:14 PM UTC
EU instance redeployment is around 10 minutes out to land production.
- monitoring Feb 02, 2026, 10:26 PM UTC
EU Courier instance deployment has landed and services operational.
- resolved Feb 02, 2026, 11:13 PM UTC
All services operational. A public-facing RFO will be available in the coming days. Thank you for your understanding and continued partnership.
- postmortem Feb 12, 2026, 06:10 PM UTC
# RFO: February 2, 2026 — Service Interruption ## Executive Summary On February 2, 2026, a deployment introduced a configuration change that referenced a file not present in our production build artifacts, causing backend services to become unavailable. API endpoints, automations, and tracking functionality were impacted for 2 hours and 17 minutes in the US region and 2 hours and 42 minutes in the Ireland region. Recovery was prolonged by a concurrent outage at GitHub Actions, our CI/CD provider, which prevented our standard automated rollback. Our team identified the root cause, executed a manual rollback independent of the affected provider, and restored full service across all regions. We have defined targeted action items to prevent recurrence. ## Incident Overview * **Affected services:** API endpoints \(including message sending\), automations, webhooks, tracking links, and authentication * **Impact:** Requests to backend services returned errors for the duration of the incident. Users were unable to send messages, trigger automations, or access tracking data. No data was lost — requests were rejected before ingestion, so no messages were partially processed or left in an inconsistent state. * **Detection:** Our monitoring systems flagged elevated error rates within minutes of the issue beginning. * **Contributing factor:** GitHub Actions, our CI/CD provider, experienced a complete outage from 10:35 to 16:30 PST. This overlapped with our incident window and prevented our standard automated rollback from executing, extending the time to resolution. ## Timeline of Events All times in PST. | **Time** | **Event** | | --- | --- | | 10:59 | Deployment of latest release initiated through standard CI/CD pipeline | | 11:44 | Deployment completed and went live; services immediately began experiencing errors due to a missing configuration dependency | | 11:49 | GitHub Actions, our CI/CD provider, experienced a complete outage, preventing standard rollback procedures | | 11:52 | Monitoring alerts triggered; engineering team engaged | | 12:00 | Incident declared; rollback initiated; engineering team assembled | | 12:07 | Status page updated — issue identified and rollback in progress. Rollback ends up being blocked by GitHub actions outage. | | 12:39 | Team pivoted to an alternative manual rollback approach. This required testing and validating the new approach. | | 13:50 | Team executed the alternative manual rollback independent of GitHub Actions after they were fully satisfied that the new approach was safe. | | 14:01 | Manual rollback completed in US region; services confirmed operational | | 14:26 | Ireland region deployment completed; services confirmed operational | | 15:13 | All services verified stable across all regions; incident resolved | ## Root Cause Analysis The disruption was traced to a configuration change included in the latest release. The change introduced a startup dependency on a utility file that was intended to be bundled with the deployment package. However, the file was not included in the production build artifacts. When backend services attempted to initialize, they were unable to locate the required file and could not start, resulting in all incoming requests being rejected. This discrepancy was not caught prior to production deployment because the file was present and functioning correctly in the development environment. The difference in how build artifacts are assembled between development and production environments meant the issue only manifested in production. ## Mitigation and Resolution 1. Upon identifying the root cause, the team immediately initiated a rollback to the prior known-good release through our standard CI/CD pipeline. 2. A complete outage at GitHub Actions, our CI/CD provider, prevented the automated rollback from completing. The team identified this external dependency and pivoted to an alternative approach. 3. The team executed a manual rollback by retrieving prior deployment artifacts from our backup storage and deploying them directly, bypassing the affected CI/CD pipeline entirely. 4. Services were restored region by region, with the US region confirmed operational at 14:01 PST and the Ireland region at 14:26 PST. 5. Extended monitoring was conducted across all services before declaring the incident fully resolved at 15:13 PST after we were satisfied everything had been stable for over 45 minutes. ## Action Items | **#** | **Action Item** | **Owner** | **Priority** | **Status** | | --- | --- | --- | --- | --- | | 1 | Require successful staging deployment and smoke tests before any production deployment | Engineering | P1 | In Progress | | 2 | Improve the reliability of automated smoke tests | Engineering | P1 | In Progress | | 3 | Add build-time validation to confirm all referenced startup dependencies are present in deployment packages | Engineering | P2 | Open | | 4 | Adopt an expedited rollback process as the standard emergency procedure, independent of CI/CD provider availability, reducing recovery time by approximately 35 minutes | Engineering | P2 | In Progress |
Minor January 29, 2026 Observability Degradation 12h 13m
Started Jan 29, 2026, 10:00 AM UTC· resolved Jan 29, 2026, 10:13 PM UTC
- resolved Jan 29, 2026, 11:13 PM UTC
The Courier team identified an issue affecting observability metrics between 21:38 UTC and 22:13 UTC, during which metrics briefly experienced an outage. A fix was released and metrics are stabilized through observability channels. Metrics received during the outage window will not be accounted for in observability dashboards.
Minor January 14, 2026 Courier Web App Performance 3h 9m Web Application
Started Jan 14, 2026, 09:47 PM UTC· resolved Jan 15, 2026, 12:56 AM UTC
- identified Jan 14, 2026, 09:47 PM UTC
The Courier team was alerted of a small degredation in the Courier web application. We have identified the issues and have rolled out a fix to mitigate. The current features will encounter some latency: - Audience Calculation - Audit Trail logging We will follow up with updates.
- resolved Jan 15, 2026, 12:56 AM UTC
Fix is out and application is stable
Critical October 20, 2025 All services impacted 51m Web Application, API, Automations, Observability and 1 more
Started Oct 20, 2025, 09:19 AM UTC· resolved Oct 20, 2025, 10:11 AM UTC
- identified Oct 20, 2025, 09:54 AM UTC
An issue with our service provider, AWS, has impacted all Courier services. The engineering team is working on resolution.
- identified Oct 20, 2025, 09:55 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Oct 20, 2025, 10:02 AM UTC
Service has been restored. Engineering is working through system review and will continue to monitor.
- resolved Oct 20, 2025, 10:11 AM UTC
The incident has been resolved
Minor October 17, 2025 Microsoft Teams Tenant Id Errors 28m API
Started Oct 17, 2025, 05:33 AM UTC· resolved Oct 17, 2025, 06:01 AM UTC
- monitoring Oct 17, 2025, 05:33 AM UTC
The Courier team was alerted of an issue with MS Teams requests experiencing errors sending. We have identified the issue and published a fix that will land in production shortly
- resolved Oct 17, 2025, 06:01 AM UTC
Release is in production and Microsoft Teams messages are passing successfully.
Notice October 13, 2025 Mailgun Message Delivery Latency –
Started Oct 13, 2025, 04:19 PM UTC· resolved Oct 13, 2025, 04:19 PM UTC
- resolved Oct 13, 2025, 04:19 PM UTC
Courier experienced a slowdown in message delivery with Mailgun providers around 2AM PST. Our monitoring systems caught the spike in the bottleneck, which eventually stabilized. Our infrastructure team is actively monitoring.
Notice October 8, 2025 Message Delivery Latency –
Started Oct 08, 2025, 09:00 AM UTC· resolved Oct 08, 2025, 09:00 AM UTC
- resolved Oct 08, 2025, 03:16 PM UTC
Courier's delivery platform experienced significant latency around 1am PST. Our platform team managed to resolve the issue and is actively monitoring for any potential issues.
Minor July 7, 2025 Webapp HTTP Interaction Issues 18m Web Application
Started Jul 07, 2025, 09:10 PM UTC· resolved Jul 07, 2025, 09:28 PM UTC
- identified Jul 07, 2025, 09:10 PM UTC
An issue with logging in to the Courier platform and overall app performance was identified by our team and a revert of the offending release has been deployed. The estimated time for the release to land is 45 minutes.
- resolved Jul 07, 2025, 09:28 PM UTC
Revert has landed and webapp is operational.
Minor June 12, 2025 Cloudflare Outage Possible Okta SSO Impact 1h 5m Web Application
Started Jun 12, 2025, 07:30 PM UTC· resolved Jun 12, 2025, 08:36 PM UTC
- monitoring Jun 12, 2025, 07:30 PM UTC
Cloudflare has reported an outage that will impact SSO features provided by WorkOS. Customers with Okta SSO might experience some degradation when signing in. Our team is monitoring the status for both Cloudflare and WorkOS. WorkOS status: https://status.workos.com/incidents/k9s870cktcsf Cloudflare Status: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
- resolved Jun 12, 2025, 08:36 PM UTC
WorkOS systems are back to normal.
Minor June 5, 2025 Message Delays on Send Pipeline 1h 17m API
Started Jun 05, 2025, 06:51 PM UTC· resolved Jun 05, 2025, 08:08 PM UTC
- investigating Jun 05, 2025, 06:51 PM UTC
The Courier team has identified an issue where there are significant delays of up to 10 minutes for messages. We are currently investigating and looking for a root cause to mitigate. Updates will follow.
- identified Jun 05, 2025, 07:04 PM UTC
Backed-up messages were caused by a bottleneck in our system, which has started to catch up. Messages will stabilize as the bottleneck clears.
- monitoring Jun 05, 2025, 07:14 PM UTC
Message bottleneck has cleared and the queue has stabilized. We will continue to monitor.
- monitoring Jun 05, 2025, 07:19 PM UTC
Messages have stabilized but we will continue to monitor.
- resolved Jun 05, 2025, 08:08 PM UTC
Messages have recovered and pipeline is stable again.
Notice May 16, 2025 EU Region – Notifications Stuck in “Queued” State 6h 34m
Started May 16, 2025, 02:28 PM UTC· resolved May 16, 2025, 09:02 PM UTC
- investigating May 16, 2025, 02:28 PM UTC
We are currently investigating an issue where EU workspace notifications appear to remain in the “queued” state.
- identified May 16, 2025, 02:53 PM UTC
The issue has been identified and a fix is being implemented.
- resolved May 16, 2025, 09:02 PM UTC
The issue has been resolved. The root cause was purely a logging issue, notifications were still sent but showing up in logs as queued. All logs have been rehydrated
Minor April 30, 2025 AWS SES Failures 6h 19m API
Started Apr 30, 2025, 10:39 PM UTC· resolved May 01, 2025, 04:58 AM UTC
- identified Apr 30, 2025, 10:39 PM UTC
The Courier team has identified an issue with a recent deployment that is impacting AWS SES integrations. We have found the root cause and have reverted the changes. The team is gathering impact.
- identified Apr 30, 2025, 10:47 PM UTC
The reverted changes will land in production in around one hour. The team will investigate the impact and handle message retries accordingly.
- monitoring Apr 30, 2025, 11:26 PM UTC
Fix has landed and the team is monitoring progress. Messages sent to AWS SES are sending normally. The team will reprocess impacted messages.
- monitoring May 01, 2025, 01:06 AM UTC
The team is still working through the reprocessing for impacted messages. Confirmation of all impacted messages reprocessed will be updated here.
- resolved May 01, 2025, 04:58 AM UTC
All impacted messages have been reprocessed.
Minor February 28, 2025 Internal Courier Error Issue 11h 22m API
Started Feb 28, 2025, 04:48 PM UTC· resolved Mar 01, 2025, 04:11 AM UTC
- identified Feb 28, 2025, 04:48 PM UTC
The Courier team has identified an issue where messages fail due to an "Internal Courier Error" in the test environment. Our team found the root cause and are working to mitigate the issue.
- identified Feb 28, 2025, 05:14 PM UTC
We have tested the fix for "Internal Courier Error" and will be releasing it to production soon.
- identified Feb 28, 2025, 05:36 PM UTC
A fix has been merged and will fully deploy within the next hour. Updates to follow.
- monitoring Feb 28, 2025, 06:40 PM UTC
The fix has landed in production and we will be monitoring for any more "Internal Courier Errors". We will be working on reprocessing any dropped messages caused by this issue.
- monitoring Feb 28, 2025, 06:57 PM UTC
Messages with tags are no longer impacted, and error has been resolved. The team is working on reprocessing dropped messages.
- monitoring Feb 28, 2025, 07:52 PM UTC
The team is still working on reprocessing dropped messages caused by this issue. Send pipeline is operational.
- resolved Mar 01, 2025, 04:11 AM UTC
All impacted messages have been reprocessed.
Minor February 21, 2025 Segment Event Processing Slowdowns 1h 15m Automations
Started Feb 21, 2025, 03:55 PM UTC· resolved Feb 21, 2025, 05:10 PM UTC
- monitoring Feb 21, 2025, 03:55 PM UTC
The Courier team has identified a slowdown in Segment event processing. There will be a slight delay for messages triggered by incoming Segment triggers. We estimate the queue to normalize soon and are monitoring.
- resolved Feb 21, 2025, 05:10 PM UTC
Segment event processing stream has resolved.
Minor February 18, 2025 Significant Notification Event Latency 24m API
Started Feb 18, 2025, 02:22 PM UTC· resolved Feb 18, 2025, 02:47 PM UTC
- identified Feb 18, 2025, 02:22 PM UTC
The Courier team identified a delay in message processing and event updating. The pipeline queue for message delivery has normalized, however significant slowdowns in message events are still processing.
- monitoring Feb 18, 2025, 02:29 PM UTC
We're monitoring our processing pipeline as messages are flowing normally. There is a slight delay in event processing which will also impact outbound webhooks.
- resolved Feb 18, 2025, 02:47 PM UTC
Message sends are flowing normally, and events are caught up without delays.
Minor February 13, 2025 Courier Send and Message Event Slowdown 36m API
Started Feb 13, 2025, 10:32 PM UTC· resolved Feb 13, 2025, 11:09 PM UTC
- identified Feb 13, 2025, 10:32 PM UTC
The Courier team has identified a slowdown in sending notifications. This issue has been identified by our team and is closely monitoring. We do not expect any messages to drop. We will continue to monitor and update accordingly.
- monitoring Feb 13, 2025, 10:51 PM UTC
Our team has identified a bottleneck in our system caused by timeouts on our sendworker. Our exponential backoff has caught up and messages are starting to go through.
- resolved Feb 13, 2025, 11:09 PM UTC
The team has identified the issue related to network connections on AWS which resulted in a backup of messages on our pipeline. Messages are flowing through as expected, and the stream has caught up.
Minor February 11, 2025 Delay in Message Sends and Processing 53m API
Started Feb 11, 2025, 10:15 PM UTC· resolved Feb 11, 2025, 11:08 PM UTC
- identified Feb 11, 2025, 10:15 PM UTC
The Courier team has identified a slowdown in our pipeline. The queue has backed up but no messages have been dropped at the moment. We will continue to update as we monitor our pipeline.
- monitoring Feb 11, 2025, 10:34 PM UTC
The team has identified the cause of message sends and slowdowns to be related to an AWS service. Messages that were stuck in a queued state are slowly passing through. The team will continue to monitor and update.
- resolved Feb 11, 2025, 11:08 PM UTC
The send stream has caught up, all messages have been sent and the issue appears to be fully resolved.
Minor January 11, 2025 Automation Delay Processing Issues 3h 2m Automations
Started Jan 11, 2025, 01:23 AM UTC· resolved Jan 11, 2025, 04:25 AM UTC
- identified Jan 11, 2025, 01:23 AM UTC
We are currently experiencing an issue affecting automation workflows that include delay steps. Some customers may encounter failures when attempting to execute automations with scheduled delays. Our engineering team has identified the root cause and is implementing a fix. We expect the service to be fully restored in a short while.
- monitoring Jan 11, 2025, 02:19 AM UTC
We have deployed a fix for the earlier issue affecting automation workflows with delay steps. New automation workflows are now executing successfully. However, automations that failed during the incident period (approximately 3:20 PM PT - 6:05 PM PT) have not yet been automatically retried. Our engineering team is monitoring the recovery and evaluating a plan to process these backlogged automations.
- resolved Jan 11, 2025, 04:25 AM UTC
The service issue affecting automation workflows with delay steps has been partially resolved. New automation workflows started after 6:05 PM PT are executing successfully. However, we have determined that automations that failed during the incident period (approximately 3:20 PM PT - 6:05 PM PT) cannot be automatically retried at this time due to technical limitations. If your business was impacted by failed automation runs during this incident, please contact our support team and we will work directly with you to address your concerns. We sincerely apologize for this disruption to your workflows.
Major October 24, 2024 Delayed Message Processing 4h 10m API
Started Oct 24, 2024, 10:38 PM UTC· resolved Oct 25, 2024, 02:48 AM UTC
- investigating Oct 24, 2024, 10:38 PM UTC
Courier has identified a delay in message processing for the messages API. We are currently investigating the root cause and will update you periodically.
- investigating Oct 24, 2024, 11:01 PM UTC
We are continuing to monitor the root cause of the delayed message processing.
- investigating Oct 24, 2024, 11:29 PM UTC
The team is still investigating and discussing the root cause internally. Messages are experiencing longer than normal processing times.
- identified Oct 25, 2024, 12:00 AM UTC
The team has identified the issue and is rolling out a fix for the message delays.
- identified Oct 25, 2024, 12:40 AM UTC
Fix has been deployed and will be live in around 1hr.
- identified Oct 25, 2024, 01:31 AM UTC
The team identified an issue with the latest fix and reverted it. A new fix has been published to mitigate the message delay issue. ETA to land in production ~45 minutes.
- monitoring Oct 25, 2024, 02:43 AM UTC
A fix has been implemented and delivery times are beginning to returning to expected levels. The team will continue to monitor.
- resolved Oct 25, 2024, 02:48 AM UTC
This incident has been resolved