Circle CI Outage History

Started Dec 12, 2025, 05:22 PM UTC· resolved Dec 12, 2025, 07:07 PM UTC

1. investigating Dec 12, 2025, 05:22 PM UTC

   We are currently investigating an issue affecting Usage API data availability for December 11, 2025. Customers querying usage data for this date will not receive results at this time. Usage data prior to December 11 remains accessible and unaffected. This issue is isolated to data retrieval for the specified date. All other CircleCI functionality, including job execution and workflow triggering, continues to operate normally. We are actively investigating the cause and will provide our next update within 30 minutes or as we know more.

2. identified Dec 12, 2025, 05:49 PM UTC

   The issue has been identified. Expected time to resolution: 90 minutes

3. resolved Dec 12, 2025, 07:07 PM UTC

   The Usage API issue has been resolved. Data is loaded for yesterday 12/11. Thank you for your patience.

Read the full incident report →

Started Dec 04, 2025, 10:53 PM UTC· resolved Dec 05, 2025, 12:15 AM UTC

1. investigating Dec 04, 2025, 10:53 PM UTC

   We're currently investigating a high number of infra failures. We'll update as soon as we know more details.

2. identified Dec 04, 2025, 11:36 PM UTC

   We are slowly returning to normal. If you had jobs that failed with "infra-fail", those jobs can be rerun. We thank you for your patience while our engineers worked to get our system back to stability.

3. monitoring Dec 05, 2025, 12:06 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 05, 2025, 12:15 AM UTC

   The incident has been resolved. Thank you for your patience. The "infra-fail" should no longer be occurring.

Read the full incident report →

Started Dec 04, 2025, 04:04 PM UTC· resolved Dec 04, 2025, 05:49 PM UTC

1. identified Dec 04, 2025, 04:04 PM UTC

   We have identified an issue in our system which mean that MacOS jobs which don't specify a resource class will fail to run with an "Invalid resource class" error. We are working to correct this, but customers can workaround this by specifying a resource class in their config.yml.

2. identified Dec 04, 2025, 04:29 PM UTC

   We have mitigated the issue and MacOS jobs which do not have a resource class specified should run normally. Customers can rerun jobs which failed an they should now run normally without any modification to their config. Thank you for your patience whillst we resolved this. Customers using M4 machine jobs may experience minor delays in jobs starting during this period as there are a large number of re-runs occurring.

3. identified Dec 04, 2025, 05:19 PM UTC

   We are continuing to see recovery, but some customers may still experience delays in jobs running on M4 executors. We appreciate your patience whilst we work through the backlog of jobs.

4. monitoring Dec 04, 2025, 05:43 PM UTC

   All jobs should now be running normally. The backlog of jobs has cleared and jobs should be running in typical time. We will continue monitoring to ensure consistent service. Thank you for your patience.

5. resolved Dec 04, 2025, 05:49 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 03, 2025, 09:31 PM UTC· resolved Dec 03, 2025, 09:58 PM UTC

1. investigating Dec 03, 2025, 09:31 PM UTC

   We are seeing some issues loading the pipelines page. This doesn't affect all users but affects the display of pipelines. We are continuing to investigate the cause of this issue.

2. investigating Dec 03, 2025, 09:31 PM UTC

   We are continuing to investigate this issue.

3. resolved Dec 03, 2025, 09:58 PM UTC

   The issues affecting the pipelines page display are related to a broader incident impacting our systems. We have opened a separate incident tracking job triggering and API status issues, which encompasses the pipelines page loading problems. Please follow https://status.circleci.com/incidents/jq4bgq2sjt1r for ongoing updates.

4. postmortem Dec 18, 2025, 09:29 PM UTC

   ## Summary _All timestamps are in UTC_ On December 2, 2025, at 16:20, we deployed a code change to workflow status computation that introduced a race condition in how workflows are terminated. This caused the following impacts: **Duplicated auto-rerun workflows \(Dec 2, 16:20 - Dec 3, 14:20\):** Some customers using auto-rerun experienced duplicate workflows being created and duplicate email notifications. **API and UI degradation \(Dec 3, 12:10 - 22:50\):** Some customers experienced issues with the Workflows API and errors in the UI when trying to view pipelines. **Job execution delays \(Dec 3, 16:20 - 16:32\):** Several customers experienced delays in job execution, with some jobs failing to start or becoming stuck. The incident was fully resolved on **December 3 at 22:50.** We thank our customers for their patience and understanding as we worked to resolve this incident. This incident tracked as multiple separate issues and status pages: * Pipelines page not loading \([Link 1](https://status.circleci.com/incidents/21cysyp8d333), [Link 2](https://status.circleci.com/incidents/fjpg0vmh476k)\) * Jobs stuck in running state \([Link](https://status.circleci.com/incidents/kvzp8x242n34)\) * Jobs not starting \([Link](https://status.circleci.com/incidents/jq4bgq2sjt1r)\) ## What Happened On December 2, 2025, at 16:20, we deployed a code change that introduced a race condition in how workflows are terminated. When workflows failed with concurrent jobs terminating, our system published duplicate workflow completion events. For some customers with auto-rerun enabled, these duplicate events triggered duplicate workflows to be created, causing cascading issues across our platform. _**Initial Detection and Response \(Dec 2, 21:40 - Dec 3, 14:20\)**_ At 21:40 on December 2nd, we received the first customer report of excessive workflow executions. Our notifications team was also alerted to a high number of notifications being sent. We initially thought it was an isolated issue related to customer automation, so we flagged the affected project at 21:46 and saw recovery on the notifications side. At 21:51 we learned that a different customer reported duplicate workflow failure emails. We suspected it might be related to a recent feature rollout, but investigation did not identify a root cause. We initially thought the elevated rate of messages from the first customer somehow impacted other customers. By 22:19, no other customer had reported similar issues and metrics were back to normal, so we decided to continue the investigation the next day. At 9:55 on December 3rd, we received alerts to elevated notifications and we resumed investigation. We escalated to a full incident at 13:44. We learned that the notifications issues seemed to be caused by auto-reruns, so we disabled auto-rerun system-wide at 13:56. We identified and rolled back the problematic code change at 14:15, then re-enabled auto-rerun at 14:20. This resolved the primary issue of duplicate workflows and notifications being created. 38 projects were impacted by the auto-rerun issue with a total of 33,135 duplicate workflows created. It also led to duplicate email notifications for 18 organizations. Because the issue manifested across multiple domains \(Orchestration, Notifications, API/UI, and Execution\), we initially investigated these as separate incidents. It took time to correlate these incidents back to a single root cause. However, this fix did not immediately resolve all customer-facing issues due to downstream impacts. ### **Downstream Impact** The excessive workflows caused cascading issues across our platform: _**Notification Issues \(Dec 2, 21:40 - Dec 3, 14:20\)**_ 18 organizations received duplicate email notifications for workflow failures. _**API and UI Issues \(Dec 3, 12:10 - 22:50\)**_ At 11:54, our monitoring detected a service queue growing due to duplicate events. While investigating, we discovered the Pipelines page was struggling to load starting at 12:10. The issue was intermittent and was initially resolved at 15:11. However, at 15:30 API errors returned and we continued to investigate. At 21:30 we identified that some pipelines had over 10,000 workflows, causing our APIs to slow down and fail. Affected customers experienced failed requests to the Workflows API and errors when trying to view pipelines. We deployed targeted fixes between 22:16 and 22:50 that filtered out problematic pipelines, restoring API performance. _**Job Execution Delays \(Dec 3, 16:20 - 16:32\)**_ At 16:18, our monitoring alerted us to issues with our job orchestration system. While investigating, we identified a drop in jobs being processed. At 16:20, our job execution infrastructure began experiencing memory pressure due to increased API traffic from the duplicate workflows. This memory pressure disrupted normal job processing and cancellation handling, and caused services to restart. Job processing recovered at 16:32. During this window, approximately 3,500 jobs were delayed, and 1,100 jobs \(3% of tasks\) failed to start. Additionally, 8,000 jobs became stuck in a canceling state. We then scaled our execution infrastructure to prevent further memory pressure. We later identified that this execution impact was connected to the API issues described above, demonstrating the cascading nature of the incident across multiple systems. ### **Resolution** All systems fully recovered by 22:50 UTC on December 3rd. We have refunded credits consumed by duplicate workflows for all 38 impacted projects on December 5th. ## Future Prevention and Process Improvement We're taking several actions to prevent similar incidents and improve our detection and response: **Preventing Duplicate Workflows** We're putting safeguards in place to prevent race conditions when workflows are terminated and we're also adding constraints to prevent duplicate auto-rerun workflows from being created in the first place. **Improved Monitoring and Alerting** While we have monitors in place, we didn't get paged for some of the issues, which delayed our incident response. As a result, we've updated existing monitoring alerts to page on-call engineers for execution infrastructure API errors and service restarts, and we're adding new alerts for UI-related issues. **Better System Resilience** We will be updating the Pipelines page to handle large numbers of workflows more gracefully while also looking to understand and improve how our systems behave under pressure. **Incident Response** One of the key challenges in this incident was the difficulty in correlating that these incidents all came from the same root cause. As a result we're taking the following actions: * Establishing better communication protocols to quickly correlate related incidents across teams * Improving coordination when mitigations in one system may impact downstream services

Read the full incident report →

Started Dec 03, 2025, 09:28 PM UTC· resolved Dec 03, 2025, 11:48 PM UTC

1. investigating Dec 03, 2025, 09:28 PM UTC

   We are currently investigating reports of jobs not starting. We apologize for the inconvenience.

2. investigating Dec 03, 2025, 09:58 PM UTC

   We are currently experiencing issues affecting job triggering and workflow starts across all resource classes. Jobs may appear stuck in a not-running state, and some customers may encounter 500 errors when making API calls to check job or workflow status. What's impacted: Job triggering, workflow starts, and API queries for job and workflow status are experiencing disruptions. This affects all resource classes and executors. Some users may also experience issues loading the pipeline page. What to expect: We are actively working to stabilize our systems and restore normal operations. We will provide updates as we make progress toward resolution. We thank you for your patience while we work through these issues - we will update with our progress within 30 minutes or earlier.

3. investigating Dec 03, 2025, 10:59 PM UTC

   We have deployed changes to mitigate the high volume of workflow queries impacting our systems. Pipeline pages that were previously failing to load are now loading successfully, and we are seeing significant reduction in API errors. What's impacted: Some customers continue to experience jobs stuck in a not-running state from earlier in the incident. New job triggering and workflow starts are now functioning normally. What's happening: We have implemented mitigation measures and increased system capacity. We are continuing to investigate the remaining stuck jobs for affected customers. What to expect: If you experienced issues loading pipeline pages or querying workflow data via the API, these should now be resolved. New jobs and workflows should trigger normally. If you have jobs that appeared stuck earlier, please try rerunning them while we continue to investigate the reports of jobs that do remain stuck for a small number of customers. The data for those workflows should be available and queryable. Next update: We will provide an update within 30 minutes. Thank you for your patience while our engineers work through this incident.

4. resolved Dec 03, 2025, 11:48 PM UTC

   We have resolved the issues affecting job triggering, workflow starts, and API queries. Our systems have been stabilized and are operating normally. What was impacted: Job triggering, workflow starts, API queries, and pipeline page loading experienced disruptions for some customers. This affected all resource classes and executors. Resolution: We implemented mitigation measures to address high volume workflow queries impacting our internal systems and increased system capacity. All new jobs and workflows are now starting normally, pipeline pages are loading, and API queries are functioning as expected. What to expect: If you have jobs that became stuck during this incident, please rerun them. If you continue to experience issues after rerunning, please contact our support team. Some customers may still see jobs stuck in a cancelling state. Engineering is aware and addressing to mitigate risk. We will continue monitoring our systems and conducting a thorough review to identify additional preventive measures.

5. postmortem Dec 18, 2025, 09:30 PM UTC

   ## Summary _All timestamps are in UTC_ On December 2, 2025, at 16:20, we deployed a code change to workflow status computation that introduced a race condition in how workflows are terminated. This caused the following impacts: **Duplicated auto-rerun workflows \(Dec 2, 16:20 - Dec 3, 14:20\):** Some customers using auto-rerun experienced duplicate workflows being created and duplicate email notifications. **API and UI degradation \(Dec 3, 12:10 - 22:50\):** Some customers experienced issues with the Workflows API and errors in the UI when trying to view pipelines. **Job execution delays \(Dec 3, 16:20 - 16:32\):** Several customers experienced delays in job execution, with some jobs failing to start or becoming stuck. The incident was fully resolved on **December 3 at 22:50.** We thank our customers for their patience and understanding as we worked to resolve this incident. This incident tracked as multiple separate issues and status pages: * Pipelines page not loading \([Link 1](https://status.circleci.com/incidents/21cysyp8d333), [Link 2](https://status.circleci.com/incidents/fjpg0vmh476k)\) * Jobs stuck in running state \([Link](https://status.circleci.com/incidents/kvzp8x242n34)\) * Jobs not starting \([Link](https://status.circleci.com/incidents/jq4bgq2sjt1r)\) ## What Happened On December 2, 2025, at 16:20, we deployed a code change that introduced a race condition in how workflows are terminated. When workflows failed with concurrent jobs terminating, our system published duplicate workflow completion events. For some customers with auto-rerun enabled, these duplicate events triggered duplicate workflows to be created, causing cascading issues across our platform. _**Initial Detection and Response \(Dec 2, 21:40 - Dec 3, 14:20\)**_ At 21:40 on December 2nd, we received the first customer report of excessive workflow executions. Our notifications team was also alerted to a high number of notifications being sent. We initially thought it was an isolated issue related to customer automation, so we flagged the affected project at 21:46 and saw recovery on the notifications side. At 21:51 we learned that a different customer reported duplicate workflow failure emails. We suspected it might be related to a recent feature rollout, but investigation did not identify a root cause. We initially thought the elevated rate of messages from the first customer somehow impacted other customers. By 22:19, no other customer had reported similar issues and metrics were back to normal, so we decided to continue the investigation the next day. At 9:55 on December 3rd, we received alerts to elevated notifications and we resumed investigation. We escalated to a full incident at 13:44. We learned that the notifications issues seemed to be caused by auto-reruns, so we disabled auto-rerun system-wide at 13:56. We identified and rolled back the problematic code change at 14:15, then re-enabled auto-rerun at 14:20. This resolved the primary issue of duplicate workflows and notifications being created. 38 projects were impacted by the auto-rerun issue with a total of 33,135 duplicate workflows created. It also led to duplicate email notifications for 18 organizations. Because the issue manifested across multiple domains \(Orchestration, Notifications, API/UI, and Execution\), we initially investigated these as separate incidents. It took time to correlate these incidents back to a single root cause. However, this fix did not immediately resolve all customer-facing issues due to downstream impacts. ### **Downstream Impact** The excessive workflows caused cascading issues across our platform: _**Notification Issues \(Dec 2, 21:40 - Dec 3, 14:20\)**_ 18 organizations received duplicate email notifications for workflow failures. _**API and UI Issues \(Dec 3, 12:10 - 22:50\)**_ At 11:54, our monitoring detected a service queue growing due to duplicate events. While investigating, we discovered the Pipelines page was struggling to load starting at 12:10. The issue was intermittent and was initially resolved at 15:11. However, at 15:30 API errors returned and we continued to investigate. At 21:30 we identified that some pipelines had over 10,000 workflows, causing our APIs to slow down and fail. Affected customers experienced failed requests to the Workflows API and errors when trying to view pipelines. We deployed targeted fixes between 22:16 and 22:50 that filtered out problematic pipelines, restoring API performance. _**Job Execution Delays \(Dec 3, 16:20 - 16:32\)**_ At 16:18, our monitoring alerted us to issues with our job orchestration system. While investigating, we identified a drop in jobs being processed. At 16:20, our job execution infrastructure began experiencing memory pressure due to increased API traffic from the duplicate workflows. This memory pressure disrupted normal job processing and cancellation handling, and caused services to restart. Job processing recovered at 16:32. During this window, approximately 3,500 jobs were delayed, and 1,100 jobs \(3% of tasks\) failed to start. Additionally, 8,000 jobs became stuck in a canceling state. We then scaled our execution infrastructure to prevent further memory pressure. We later identified that this execution impact was connected to the API issues described above, demonstrating the cascading nature of the incident across multiple systems. ### **Resolution** All systems fully recovered by 22:50 UTC on December 3rd. We have refunded credits consumed by duplicate workflows for all 38 impacted projects on December 5th. ## Future Prevention and Process Improvement We're taking several actions to prevent similar incidents and improve our detection and response: **Preventing Duplicate Workflows** We're putting safeguards in place to prevent race conditions when workflows are terminated and we're also adding constraints to prevent duplicate auto-rerun workflows from being created in the first place. **Improved Monitoring and Alerting** While we have monitors in place, we didn't get paged for some of the issues, which delayed our incident response. As a result, we've updated existing monitoring alerts to page on-call engineers for execution infrastructure API errors and service restarts, and we're adding new alerts for UI-related issues. **Better System Resilience** We will be updating the Pipelines page to handle large numbers of workflows more gracefully while also looking to understand and improve how our systems behave under pressure. **Incident Response** One of the key challenges in this incident was the difficulty in correlating that these incidents all came from the same root cause. As a result we're taking the following actions: * Establishing better communication protocols to quickly correlate related incidents across teams * Improving coordination when mitigations in one system may impact downstream services

Read the full incident report →

Started Dec 03, 2025, 04:53 PM UTC· resolved Dec 03, 2025, 06:12 PM UTC

1. investigating Dec 03, 2025, 04:53 PM UTC

   At 16:20 UTC, we began experiencing delays in job triggering and starts across all resource classes. Some workflows and jobs may appear stuck in a running state. What’s impacted: Job triggering is experiencing delays or is stuck. This affects all resource classes and executors. What to expect: If you have workflows that appear stuck and haven’t started, we recommend manually rerunning them. We are actively investigating the root cause and working to restore normal processing speeds. Next update: We will provide an update within 30 minutes or earlier with our progress

2. monitoring Dec 03, 2025, 05:27 PM UTC

   As of 16:32 UTC, job triggering and workflow starts have returned to normal operation across all resource classes. The impact was limited to a 12-minute window between 16:20 and 16:32 UTC. What's impacted: All new jobs and workflows are now starting normally. What to expect: If you have workflows or jobs that were stuck during the 16:20-16:32 UTC window, please manually rerun them. We are continuing to investigate the root cause of this disruption and will provide an update within 30 minutes or once our investigation is complete.

3. resolved Dec 03, 2025, 06:12 PM UTC

   Between 16:20 and 16:32 UTC, job triggering and workflow starts experienced disruptions across all resource classes due to memory pressure on our internal job distributor systems. We identified the issue and scaled our infrastructure to handle the load. Services returned to normal operation at 16:32 UTC. What was impacted: Job triggering and workflow starts were disrupted for 12 minutes. Some workflows and jobs appeared stuck in a running state during this window. Resolution: Our systems are now operating normally with additional capacity in place to prevent similar disruptions. If you had workflows or jobs that were stuck during this window, please manually rerun them. The incident is now resolved and we will be conducting a thorough review to understand what triggered the memory pressure and identify any additional preventive measures

4. postmortem Dec 18, 2025, 09:30 PM UTC

   ## Summary _All timestamps are in UTC_ On December 2, 2025, at 16:20, we deployed a code change to workflow status computation that introduced a race condition in how workflows are terminated. This caused the following impacts: **Duplicated auto-rerun workflows \(Dec 2, 16:20 - Dec 3, 14:20\):** Some customers using auto-rerun experienced duplicate workflows being created and duplicate email notifications. **API and UI degradation \(Dec 3, 12:10 - 22:50\):** Some customers experienced issues with the Workflows API and errors in the UI when trying to view pipelines. **Job execution delays \(Dec 3, 16:20 - 16:32\):** Several customers experienced delays in job execution, with some jobs failing to start or becoming stuck. The incident was fully resolved on **December 3 at 22:50.** We thank our customers for their patience and understanding as we worked to resolve this incident. This incident tracked as multiple separate issues and status pages: * Pipelines page not loading \([Link 1](https://status.circleci.com/incidents/21cysyp8d333), [Link 2](https://status.circleci.com/incidents/fjpg0vmh476k)\) * Jobs stuck in running state \([Link](https://status.circleci.com/incidents/kvzp8x242n34)\) * Jobs not starting \([Link](https://status.circleci.com/incidents/jq4bgq2sjt1r)\) ## What Happened On December 2, 2025, at 16:20, we deployed a code change that introduced a race condition in how workflows are terminated. When workflows failed with concurrent jobs terminating, our system published duplicate workflow completion events. For some customers with auto-rerun enabled, these duplicate events triggered duplicate workflows to be created, causing cascading issues across our platform. _**Initial Detection and Response \(Dec 2, 21:40 - Dec 3, 14:20\)**_ At 21:40 on December 2nd, we received the first customer report of excessive workflow executions. Our notifications team was also alerted to a high number of notifications being sent. We initially thought it was an isolated issue related to customer automation, so we flagged the affected project at 21:46 and saw recovery on the notifications side. At 21:51 we learned that a different customer reported duplicate workflow failure emails. We suspected it might be related to a recent feature rollout, but investigation did not identify a root cause. We initially thought the elevated rate of messages from the first customer somehow impacted other customers. By 22:19, no other customer had reported similar issues and metrics were back to normal, so we decided to continue the investigation the next day. At 9:55 on December 3rd, we received alerts to elevated notifications and we resumed investigation. We escalated to a full incident at 13:44. We learned that the notifications issues seemed to be caused by auto-reruns, so we disabled auto-rerun system-wide at 13:56. We identified and rolled back the problematic code change at 14:15, then re-enabled auto-rerun at 14:20. This resolved the primary issue of duplicate workflows and notifications being created. 38 projects were impacted by the auto-rerun issue with a total of 33,135 duplicate workflows created. It also led to duplicate email notifications for 18 organizations. Because the issue manifested across multiple domains \(Orchestration, Notifications, API/UI, and Execution\), we initially investigated these as separate incidents. It took time to correlate these incidents back to a single root cause. However, this fix did not immediately resolve all customer-facing issues due to downstream impacts. ### **Downstream Impact** The excessive workflows caused cascading issues across our platform: _**Notification Issues \(Dec 2, 21:40 - Dec 3, 14:20\)**_ 18 organizations received duplicate email notifications for workflow failures. _**API and UI Issues \(Dec 3, 12:10 - 22:50\)**_ At 11:54, our monitoring detected a service queue growing due to duplicate events. While investigating, we discovered the Pipelines page was struggling to load starting at 12:10. The issue was intermittent and was initially resolved at 15:11. However, at 15:30 API errors returned and we continued to investigate. At 21:30 we identified that some pipelines had over 10,000 workflows, causing our APIs to slow down and fail. Affected customers experienced failed requests to the Workflows API and errors when trying to view pipelines. We deployed targeted fixes between 22:16 and 22:50 that filtered out problematic pipelines, restoring API performance. _**Job Execution Delays \(Dec 3, 16:20 - 16:32\)**_ At 16:18, our monitoring alerted us to issues with our job orchestration system. While investigating, we identified a drop in jobs being processed. At 16:20, our job execution infrastructure began experiencing memory pressure due to increased API traffic from the duplicate workflows. This memory pressure disrupted normal job processing and cancellation handling, and caused services to restart. Job processing recovered at 16:32. During this window, approximately 3,500 jobs were delayed, and 1,100 jobs \(3% of tasks\) failed to start. Additionally, 8,000 jobs became stuck in a canceling state. We then scaled our execution infrastructure to prevent further memory pressure. We later identified that this execution impact was connected to the API issues described above, demonstrating the cascading nature of the incident across multiple systems. ### **Resolution** All systems fully recovered by 22:50 UTC on December 3rd. We have refunded credits consumed by duplicate workflows for all 38 impacted projects on December 5th. ## Future Prevention and Process Improvement We're taking several actions to prevent similar incidents and improve our detection and response: **Preventing Duplicate Workflows** We're putting safeguards in place to prevent race conditions when workflows are terminated and we're also adding constraints to prevent duplicate auto-rerun workflows from being created in the first place. **Improved Monitoring and Alerting** While we have monitors in place, we didn't get paged for some of the issues, which delayed our incident response. As a result, we've updated existing monitoring alerts to page on-call engineers for execution infrastructure API errors and service restarts, and we're adding new alerts for UI-related issues. **Better System Resilience** We will be updating the Pipelines page to handle large numbers of workflows more gracefully while also looking to understand and improve how our systems behave under pressure. **Incident Response** One of the key challenges in this incident was the difficulty in correlating that these incidents all came from the same root cause. As a result we're taking the following actions: * Establishing better communication protocols to quickly correlate related incidents across teams * Improving coordination when mitigations in one system may impact downstream services

Read the full incident report →

Started Dec 03, 2025, 01:37 PM UTC· resolved Dec 03, 2025, 03:30 PM UTC

1. identified Dec 03, 2025, 01:37 PM UTC

   We are seeing some issues loading the pipelines page. This is intermittent and won't affect most users. No work is being affected, just the display of pipelines. We have identified the issue and are working on a fix.

2. resolved Dec 03, 2025, 03:30 PM UTC

   This incident has been resolved. Things should be back to normal.

3. postmortem Dec 18, 2025, 09:28 PM UTC

   ## Summary _All timestamps are in UTC_ On December 2, 2025, at 16:20, we deployed a code change to workflow status computation that introduced a race condition in how workflows are terminated. This caused the following impacts: **Duplicated auto-rerun workflows \(Dec 2, 16:20 - Dec 3, 14:20\):** Some customers using auto-rerun experienced duplicate workflows being created and duplicate email notifications. **API and UI degradation \(Dec 3, 12:10 - 22:50\):** Some customers experienced issues with the Workflows API and errors in the UI when trying to view pipelines. **Job execution delays \(Dec 3, 16:20 - 16:32\):** Several customers experienced delays in job execution, with some jobs failing to start or becoming stuck. The incident was fully resolved on **December 3 at 22:50.** We thank our customers for their patience and understanding as we worked to resolve this incident. This incident tracked as multiple separate issues and status pages: * Pipelines page not loading \([Link 1](https://status.circleci.com/incidents/21cysyp8d333), [Link 2](https://status.circleci.com/incidents/fjpg0vmh476k)\) * Jobs stuck in running state \([Link](https://status.circleci.com/incidents/kvzp8x242n34)\) * Jobs not starting \([Link](https://status.circleci.com/incidents/jq4bgq2sjt1r)\) ## What Happened On December 2, 2025, at 16:20, we deployed a code change that introduced a race condition in how workflows are terminated. When workflows failed with concurrent jobs terminating, our system published duplicate workflow completion events. For some customers with auto-rerun enabled, these duplicate events triggered duplicate workflows to be created, causing cascading issues across our platform. _**Initial Detection and Response \(Dec 2, 21:40 - Dec 3, 14:20\)**_ At 21:40 on December 2nd, we received the first customer report of excessive workflow executions. Our notifications team was also alerted to a high number of notifications being sent. We initially thought it was an isolated issue related to customer automation, so we flagged the affected project at 21:46 and saw recovery on the notifications side. At 21:51 we learned that a different customer reported duplicate workflow failure emails. We suspected it might be related to a recent feature rollout, but investigation did not identify a root cause. We initially thought the elevated rate of messages from the first customer somehow impacted other customers. By 22:19, no other customer had reported similar issues and metrics were back to normal, so we decided to continue the investigation the next day. At 9:55 on December 3rd, we received alerts to elevated notifications and we resumed investigation. We escalated to a full incident at 13:44. We learned that the notifications issues seemed to be caused by auto-reruns, so we disabled auto-rerun system-wide at 13:56. We identified and rolled back the problematic code change at 14:15, then re-enabled auto-rerun at 14:20. This resolved the primary issue of duplicate workflows and notifications being created. 38 projects were impacted by the auto-rerun issue with a total of 33,135 duplicate workflows created. It also led to duplicate email notifications for 18 organizations. Because the issue manifested across multiple domains \(Orchestration, Notifications, API/UI, and Execution\), we initially investigated these as separate incidents. It took time to correlate these incidents back to a single root cause. However, this fix did not immediately resolve all customer-facing issues due to downstream impacts. ### **Downstream Impact** The excessive workflows caused cascading issues across our platform: _**Notification Issues \(Dec 2, 21:40 - Dec 3, 14:20\)**_ 18 organizations received duplicate email notifications for workflow failures. _**API and UI Issues \(Dec 3, 12:10 - 22:50\)**_ At 11:54, our monitoring detected a service queue growing due to duplicate events. While investigating, we discovered the Pipelines page was struggling to load starting at 12:10. The issue was intermittent and was initially resolved at 15:11. However, at 15:30 API errors returned and we continued to investigate. At 21:30 we identified that some pipelines had over 10,000 workflows, causing our APIs to slow down and fail. Affected customers experienced failed requests to the Workflows API and errors when trying to view pipelines. We deployed targeted fixes between 22:16 and 22:50 that filtered out problematic pipelines, restoring API performance. _**Job Execution Delays \(Dec 3, 16:20 - 16:32\)**_ At 16:18, our monitoring alerted us to issues with our job orchestration system. While investigating, we identified a drop in jobs being processed. At 16:20, our job execution infrastructure began experiencing memory pressure due to increased API traffic from the duplicate workflows. This memory pressure disrupted normal job processing and cancellation handling, and caused services to restart. Job processing recovered at 16:32. During this window, approximately 3,500 jobs were delayed, and 1,100 jobs \(3% of tasks\) failed to start. Additionally, 8,000 jobs became stuck in a canceling state. We then scaled our execution infrastructure to prevent further memory pressure. We later identified that this execution impact was connected to the API issues described above, demonstrating the cascading nature of the incident across multiple systems. ### **Resolution** All systems fully recovered by 22:50 UTC on December 3rd. We have refunded credits consumed by duplicate workflows for all 38 impacted projects on December 5th. ## Future Prevention and Process Improvement We're taking several actions to prevent similar incidents and improve our detection and response: **Preventing Duplicate Workflows** We're putting safeguards in place to prevent race conditions when workflows are terminated and we're also adding constraints to prevent duplicate auto-rerun workflows from being created in the first place. **Improved Monitoring and Alerting** While we have monitors in place, we didn't get paged for some of the issues, which delayed our incident response. As a result, we've updated existing monitoring alerts to page on-call engineers for execution infrastructure API errors and service restarts, and we're adding new alerts for UI-related issues. **Better System Resilience** We will be updating the Pipelines page to handle large numbers of workflows more gracefully while also looking to understand and improve how our systems behave under pressure. **Incident Response** One of the key challenges in this incident was the difficulty in correlating that these incidents all came from the same root cause. As a result we're taking the following actions: * Establishing better communication protocols to quickly correlate related incidents across teams * Improving coordination when mitigations in one system may impact downstream services

Read the full incident report →

Started Dec 02, 2025, 02:00 AM UTC· resolved Dec 02, 2025, 02:00 AM UTC

1. resolved Dec 03, 2025, 04:01 PM UTC

   Between Dec 2, 2025 16:20 UTC and Dec 3, 2025 14:27 UTC, a change deployed to our workflow execution system caused duplicate notifications to be sent to some customers and triggered unexpected auto-reruns for a small number of projects. Impact: - Some customers received multiple failure notification emails for the same workflow. - The number of duplicate notifications varied based on how many jobs in the workflow were affected (e.g., marked skipped or cancelled). - A subset of customers may have received duplicate Slack notifications. - A small number of projects experienced duplicated auto-reruns. The issue has been fully mitigated, and notification behavior has returned to normal. We appreciate your patience while our team resolved this issue.

Read the full incident report →

Started Nov 26, 2025, 10:53 PM UTC· resolved Nov 26, 2025, 11:24 PM UTC

1. identified Nov 26, 2025, 10:53 PM UTC

   We are seeing an increase in wait times to start jobs on Mac M1 Large. Customers are encouraged to move workloads to Mac M4 if possible.

2. resolved Nov 26, 2025, 11:24 PM UTC

   We are in the process of deprecating our M1 fleet due to limited machine capacity. So while all jobs will run, some running on M1 Large may be delayed. Customers may move to M1 Medium if you must have an M1 chip. Otherwise we encourage customers to move to M2 and M4 resource classes. Thank you for patience and we apologize for any inconvenience.

Read the full incident report →

Started Nov 26, 2025, 03:50 PM UTC· resolved Nov 26, 2025, 04:28 PM UTC

1. investigating Nov 26, 2025, 03:50 PM UTC

   We are currently experiencing issues with certain resource classes for Machine Executors.

2. monitoring Nov 26, 2025, 04:06 PM UTC

   We've identified the issue, and have rolled out a fix. We are now monitoring. To our affected customers, thank you for your patience as we got this resolved.

3. resolved Nov 26, 2025, 04:28 PM UTC

   All Machine resource classes are operating successfully. Thank you for your patience, and we apologize for any inconvenience this may have caused.

Read the full incident report →

Started Nov 25, 2025, 04:10 PM UTC· resolved Nov 25, 2025, 04:10 PM UTC

1. resolved Nov 25, 2025, 04:30 PM UTC

   There was a short issue from 16:10 UTC to 16:13 UTC which meant that we did not successfully trigger some jobs. If you expected a job to run which did not within the last 30 minutes you should attempt to run the job manually or attempt to trigger the job again. (For example, by pushing to your repository).

Read the full incident report →

Started Nov 22, 2025, 01:00 AM UTC· resolved Nov 22, 2025, 01:00 AM UTC

1. resolved Nov 22, 2025, 01:15 AM UTC

   Between 20:20 and 20:26 UTC, following our scheduled RabbitMQ maintenance, a small number of workflows may have experienced issues with GitHub check notifications. Impact: - Some workflows showed no check run status in GitHub - Some workflows showed check runs stuck in "pending" state - User flows requiring these checks may have been blocked Resolution: This issue resolved upon completion of our maintenance. All services are now operating normally. We apologize for any inconvenience during this maintenance period. If you have builds that completed during this window with pending or missing checks, a rerun will restore the correct status.

Read the full incident report →

Started Nov 20, 2025, 03:58 PM UTC· resolved Nov 20, 2025, 04:08 PM UTC

1. investigating Nov 20, 2025, 03:58 PM UTC

   We are investigating delays in M4 and M1 Mac jobs, and are working on resolving this as soon as possible. Thank you for your patience.

2. resolved Nov 20, 2025, 04:08 PM UTC

   Mac M4 and M1 jobs are now running normally. Thank you again for your patience while we addressed this issue.

Read the full incident report →

Started Nov 19, 2025, 04:21 PM UTC· resolved Nov 19, 2025, 05:19 PM UTC

1. investigating Nov 19, 2025, 04:21 PM UTC

   CircleCI UI is currently not loading. We are investigating, but jobs are running normally.

2. monitoring Nov 19, 2025, 04:26 PM UTC

   We have applied a fix which has brought the UI back online. We are monitoring for further issues.

3. monitoring Nov 19, 2025, 05:19 PM UTC

   This incident has been resolved. The CircleCI UI is not loading and operating normally for all customers. Thank you for your patience, and we apologize for any inconvenience this may have caused.

4. resolved Nov 19, 2025, 05:19 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 18, 2025, 04:44 PM UTC· resolved Nov 18, 2025, 07:57 PM UTC

1. identified Nov 18, 2025, 04:44 PM UTC

   We are currently experiencing capacity constraints affecting Linux, AMD, ARM and Remote Docker resource classes, resulting in job start delays of up to 3 minutes. Current Status: - Linux, AMD, ARM and Remote Docker jobs: Experiencing delays up to 3 minutes due to capacity constraints - Docker, Mac, Windows, and Android jobs: Operating normally without delays Our engineering team is actively working to address these constraints and expand available capacity. All jobs will continue to run normally after the initial delay. We appreciate your patience as we work to resolve this issue. Next Update: Within 30 minutes or as the situation changes.

2. identified Nov 18, 2025, 05:19 PM UTC

   We continue to work on addressing the capacity constraints affecting Linux, ARM, AMD, and Remote Docker resource classes. Job start delays remain stable at approximately 3 minutes. Current Status: - Linux, ARM, AMD and Remote Docker jobs: Delays holding steady at up to 3 minutes - Docker, Mac, Windows, and Android jobs: Operating normally without delays Our team is actively pursuing multiple approaches to expand capacity and reduce these delays. All jobs continue to run successfully after the initial wait time. Thank you for your continued patience as we work toward resolution. Next Update: Within 30 minutes or when conditions change.

3. identified Nov 18, 2025, 06:09 PM UTC

   We're seeing improvements in job start times for Linux, ARM, AMD, and Remote Docker resource classes. We are actively working on a stable solution to address this issue. Current Status: - Linux, ARM, AMD and Remote Docker jobs: Wait times have improved, though some delays may still occur - Docker, Mac, Windows, and Android jobs: Operating normally Our engineering team continues to work on implementing additional capacity and solutions to prevent future constraints. While current performance has improved, we're keeping this open as we work toward a stable resolution. Thank you for your patience as we address the underlying capacity challenges. Next Update: Within 30 minutes or as conditions change.

4. monitoring Nov 18, 2025, 07:10 PM UTC

   Task wait times have returned to normal for all resource classes including Linux, ARM, AMD, and Remote Docker. We are monitoring system stability before marking this incident as fully resolved. Current Status: - All resource classes: Operating with normal queue times - System performance: Stable and meeting expected levels - Monitoring: Actively watching for any signs of recurring capacity constraints We will continue monitoring for the next 30 minutes to ensure the improvements are sustained before closing this incident. Thank you for your patience during this disruption. Next Update: Within 30 minutes or when resolved.

5. resolved Nov 18, 2025, 07:57 PM UTC

   This incident has been resolved. All resource classes including Linux, ARM, AMD, and Remote Docker are operating normally with standard queue times. System performance has been stable and capacity has been restored. Thank you for your patience while our engineers addressed this issue

Read the full incident report →

Started Nov 17, 2025, 04:18 PM UTC· resolved Nov 17, 2025, 08:23 PM UTC

1. investigating Nov 17, 2025, 04:18 PM UTC

   We are investigating Machine and Remote Docker delays. We are working on resolving this as quickly as possible. All jobs will continue to run normally after the delay. Thank you for your patience.

2. identified Nov 17, 2025, 05:00 PM UTC

   We have identified the cause of Machine and Remote Docker delays. We are making changes which we expect to reduce the delays. All jobs will continue to run normally after the delay, and we expect the delay will decrease with time. Thank you for your patience.

3. identified Nov 17, 2025, 06:46 PM UTC

   We are actively adding additional capacity to reduce job start delays for Linux , AMD and Remote Docker resource classes. Current improvements show average wait times are now under one minute for most jobs, with some customers waiting up to 6 minutes for jobs using these resources. . Current Status: - Linux, AMD and Remote Docker jobs: Wait times improving as we expand capacity, delays up to 6 minutes - Docker, Mac, Windows, and Android jobs: Operating normally without delays Our team is continuing to deploy additional resources to fully eliminate the remaining delays. We expect continued improvement over the next hour as this expanded capacity comes online. We appreciate your patience as we optimize our infrastructure to meet current demand. Next Update: Within 30 minutes or when normal operation is restored.

4. monitoring Nov 17, 2025, 07:51 PM UTC

   We continue to see reductions in average wait times for customers requesting Linux, AMD, and Remote Docker resource classes, with maximum delays now reduced to under 3 minutes. Current Status: - Linux, AMD and Remote Docker jobs: Wait times continue to improve, maximum delays of just under 3 minutes - Docker, Mac, Windows, and Android jobs: Operating normally without delays Additional capacity continues to come online, and we expect further improvements as our infrastructure scales to meet demand. Thank you for your patience as we work toward full resolution. Next Update: Within 30 minutes or when normal operation is restored.

5. resolved Nov 17, 2025, 08:23 PM UTC

   This incident has been resolved. All resource classes including Linux, AMD, and Remote Docker are now operating normally with standard queue times. The additional capacity has been successfully deployed and job start delays have been eliminated. Thank you for your patience while our engineers worked through this issue.

Read the full incident report →

Started Nov 12, 2025, 05:38 PM UTC· resolved Nov 12, 2025, 05:56 PM UTC

1. identified Nov 12, 2025, 05:38 PM UTC

   There are currently issues creating and modifying projects using the Project Settings UI. We have identified the cause and are working on a remediation now. We appreciate your patience whilst we work to resolve an issue.

2. resolved Nov 12, 2025, 05:56 PM UTC

   We have resolved the issue which prevented users from creating and modifying projects. Any attempts to create projects will need to be retried. Thank you for your patenience whilst we resolved it.

Read the full incident report →

Started Nov 07, 2025, 10:40 AM UTC· resolved Nov 10, 2025, 11:24 AM UTC

1. monitoring Nov 07, 2025, 10:40 AM UTC

   At present customers running Docker containers cannot use /mnt/ramdisk. Customers will need to modify their configuration to use the filesystem instead. This has occurred in order to update the Docker Infrastructure that supports our `working_directory: /mnt/ramdisk` feature. This is a necessary step in order to address CVE-2025-31133, CVE-2025-52565, and CVE-2025-52881. This bug affects any Docker jobs utilizing `working_directory: /mnt/ramdisk`. Jobs will fail on checkout and workspace attachments. The immediate workaround will be to remove any usage of `working_directory: /mnt/ramdisk` from your job which will require an update of `.circleci/config.yml` Once this maintenance is complete, customers will be unable to utilize `working_directory: /mnt/ramdisk`until further notice.

2. resolved Nov 10, 2025, 11:24 AM UTC

   We have identified and applied a workaround for the RAM DIsk issue, which means customers can resume builds using `/mnt/ramdisk`. We appreciate your patience and understanding whilst we worked on resolving this whilst ensuring the security of our platform for all customers.

Read the full incident report →

Started Nov 06, 2025, 07:26 PM UTC· resolved Nov 06, 2025, 07:26 PM UTC

1. resolved Nov 06, 2025, 07:26 PM UTC

   Some customers from 15:26:06 UTC to 18:44:16 UTC would have experienced intermittent issues with running docker jobs. We have since rolled back this change.

Read the full incident report →

Started Nov 05, 2025, 04:57 PM UTC· resolved Nov 05, 2025, 06:59 PM UTC

1. identified Nov 05, 2025, 04:57 PM UTC

   We are experiencing delays starting jobs for M1 and M4 medium and large resource classes. Running jobs on these resource classes will also experience delays due to degraded network performance.

2. identified Nov 05, 2025, 05:30 PM UTC

   We are continuing to investigate this and are working with our datacenter provider to mitigate this issue.

3. identified Nov 05, 2025, 06:06 PM UTC

   We are continuing to investigate degraded network performance affecting macOS resources in our infrastructure. What's happening: We have identified reduced network bandwidth that is causing performance degradation for specific macOS resource classes. What's impacted: M1 medium and large resources: Experiencing average wait times of up to 20 minutes before job start M4 Pro medium and large resources: Experiencing average wait times of approximately 5 minutes before job start Running jobs on affected resources: Network-dependent operations within jobs (cache restoration, pulling datasets, downloading dependencies) are also experiencing slowdowns. All other macOS resource classes are operating normally Next update: We are actively working to resolve this issue and will provide an update within 30 minutes or sooner if the situation changes.

4. monitoring Nov 05, 2025, 06:41 PM UTC

   We are observing recovery of the macOS resources performance issue. Wait times for M1 medium/large and M4 Pro medium/large resources have returned to normal levels Network performance within running jobs appears to be recovering We are continuing to monitor the situation closely to ensure stability and will provide another update within 30 minutes as we confirm full resolution of the incident.

5. resolved Nov 05, 2025, 06:59 PM UTC

   The macOS resources performance issue has been fully resolved. All M1 and M4 Pro medium/large resources are operating normally with standard wait times and network performance. We have monitored the infrastructure for stability and confirmed full recovery of all affected services. We apologize for any inconvenience this may have caused. Thank you for your patience during this incident.

Read the full incident report →

Started Nov 04, 2025, 11:32 PM UTC· resolved Nov 05, 2025, 12:16 AM UTC

1. investigating Nov 04, 2025, 11:32 PM UTC

   We are currently investigating an issue preventing users from adding User Keys to a project when clicking on "Authorize with Github". Adding any other keys such as deploy keys or existing authorized user keys continues to function normally.

2. resolved Nov 05, 2025, 12:16 AM UTC

   We have identified a workaround for the issue preventing users from adding User Keys via the "Authorize with GitHub" option. While the "Authorize with GitHub" button is currently experiencing issues, users can successfully add SSH keys to their projects using our fully supported and documented manual process : https://circleci.com/docs/guides/integration/add-ssh-key/#steps-to-add-additional-ssh-keys This workaround provides full functionality for adding SSH keys to your projects, effectively resolving the immediate impact. Our engineering team is actively monitoring this issue and will implement a permanent fix to restore the GitHub authorization flow. Post resolution Update - 17:30 UTC The GitHub authorization functionality for adding User Keys has been fully restored as of 9:21 UTC. The "Authorize with GitHub" button is now working as expected in project settings. The manual workaround previously provided is no longer necessary, though it remains a valid alternative method for adding SSH keys. Thank you for your patience while we resolved this issue.

Read the full incident report →

Started Oct 27, 2025, 03:04 PM UTC· resolved Oct 27, 2025, 03:24 PM UTC

1. investigating Oct 27, 2025, 03:04 PM UTC

   There is an issue with Linux jobs not running and we are investigating.

2. identified Oct 27, 2025, 03:10 PM UTC

   We have identified the issue, it has been mitigated and we are waiting for jobs to catch. No work was lost, just slight delays happened.

3. monitoring Oct 27, 2025, 03:17 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 27, 2025, 03:24 PM UTC

   This incident has been resolved. Thank you for your patience.

Read the full incident report →

Started Oct 20, 2025, 07:49 AM UTC· resolved Oct 20, 2025, 10:15 PM UTC

1. investigating Oct 20, 2025, 07:49 AM UTC

   We are currently experiencing an issue with an upstream service provider that is causing builds to fail or experience delays in the queue. We will provide updates as more information becomes available.

2. investigating Oct 20, 2025, 09:10 AM UTC

   Our upstream service provider has identified the root cause and is actively working on mitigation. We'll continue monitoring and provide updates as more information becomes available.

3. identified Oct 20, 2025, 12:06 PM UTC

   We continue to see delays in getting instances to run jobs from AWS due to their continued incident https://health.aws.amazon.com/health/status. This is causing delays in jobs starting across Docker and Linux.

4. identified Oct 20, 2025, 02:40 PM UTC

   We continue to see delays in getting instances from AWS (https://health.aws.amazon.com/health/status) and are actively monitoring the situation.

5. identified Oct 20, 2025, 03:45 PM UTC

   We continue to experience delays in acquiring new instances from AWS (https://health.aws.amazon.com/health/status) and are actively monitoring recovery. In addition, we’re investigating issues affecting macOS M4Pro jobs where a critical network service is intermittently failing, causing an increased queue time. Our teams are working to mitigate impact and will provide updates as we learn more.

6. identified Oct 20, 2025, 05:01 PM UTC

   We are continuing to experience high levels of errors attempting to run instances in AWS (https://health.aws.amazon.com/health/status). Due to this, we are currently unable to run Linux Machine and Remote Docker jobs, and Docker is experiencing slow scaling. Additionally, AWS network instability prevents us from booting our MacOS M4 fleet. Customers attempting to trigger Docker jobs will see queueing with slow progress. Windows and Android jobs are unaffected.

7. identified Oct 20, 2025, 06:31 PM UTC

   We continue to experience issues running Linux machine and Remote Docker jobs due to capacity issues at AWS following their recent incident (https://health.aws.amazon.com/health/status). Docker and Mac job performance has recovered to normal levels, and Windows and Android jobs are still unaffected.

8. identified Oct 20, 2025, 08:06 PM UTC

   We continue to experience capacity limitations for Linux machine and Remote Docker jobs related to the AWS incident (https://health.aws.amazon.com/health/status). Docker, Mac, Windows and Android jobs are all operating normally. We're actively monitoring the situation and will continue to provide updates as the situation evolves. We appreciate your patience as we work through these infrastructure constraints.

9. identified Oct 20, 2025, 09:08 PM UTC

   We're seeing meaningful improvement in Linux machine and Remote Docker job performance as AWS recovers from their incident (https://health.aws.amazon.com/health/status). Current Status: Linux machine and Remote Docker jobs: Average wait times have improved to approximately 10-15 minutes Docker and Mac jobs: Operating normally though ARM and IP-Ranges Docker jobs may still experience longer queue times Windows and Android jobs: Unaffected While we haven't returned to full capacity, the situation is steadily improving. We continue to work and monitor the queue times closely. Thank you for your patience during this incident. We'll continue to update you as conditions improve.

10. monitoring Oct 20, 2025, 09:42 PM UTC

    Queue times for all job types have returned to normal as AWS continues to recover from their incident (https://health.aws.amazon.com/health/status). All job types: Operating at normal capacity with standard queue times Performance has fully restored across Linux machine, Remote Docker, Docker, ARM, IP-Ranges Docker, and Mac jobs. Windows and Android were unaffected throughout the incident. We're continuing to monitor system performance closely to ensure stability. Thank you for your patience during this incident.

11. resolved Oct 20, 2025, 10:15 PM UTC

    This incident has been resolved. All job types are now operating normally with standard queue times. Linux machine and Remote Docker capacity has been fully restored following the AWS incident recovery. Thank you for your patience.

12. postmortem Nov 05, 2025, 02:57 PM UTC

    ## Summary On October 20, 2025, from 06:55 to 22:00 UTC, CircleCI customers experienced a variety of failures and delays across the system including job failures, pipeline creation failures, periods of increased queueing, errors in the UI and APIs, and a period of unavailability for audit log streaming failures due to an AWS outage affecting multiple AWS services in the US-EAST-1 region. We want to thank our customers for your patience and understanding as we worked to resolve this incident. The original status page for this incident can be found [here](https://status.circleci.com/incidents/0y7q12npsl4h). ## What Happened \(All times UTC\) On October 20, 2025 at 06:55 the system that handles requests to create new pipelines \(builds service\) began to receive `503 Service Unavailable` error responses when attempting to retrieve pipeline values from AWS S3, resulting in failures and delays across multiple executor types. At 07:11, [AWS declared an incident](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_BA540_514A652BE1A) indicating that multiple AWS services were experiencing increased error rates and latencies in the US-EAST-1 region. During the same time, CircleCI customers began experiencing premature failures on running jobs across multiple executors. Audit log access and streaming was also unavailable at this time. CircleCI declared an incident at 07:29. At 07:40, builds service started to encounter errors when attempting to read and write from S3. This resulted in errors loading the pipelines page in the CircleCI UI as well as errors creating pipelines. At the same time, the service responsible for starting workflows also encountered issues accessing S3, resulting in some jobs being dropped. Pipeline creation slowly began to recover at 08:24, and completely recovered by 09:14 along with workflow runs. During this time, Docker, Mac M1 and M4, Windows, and Android jobs began to run again. Linux, Remote Docker, and Mac M2 jobs continued to queue. At 09:30, the system responsible for provisioning machines was unable to start new instances in AWS EC2 to meet the growing backlog of customer jobs. AWS acknowledged the increase in error rates when launching EC2 instances at 10:35. Docker jobs continued to run, but customers experienced high queue times as demand grew. By 12:20, the queue time for Docker jobs dropped to 18 minutes from a previous high of 50 minutes. The queue of Docker jobs continued to shrink and eventually clear by 14:00. At 14:30, CircleCI engineers made a change to prevent infrastructure responsible for processing Docker jobs from scaling down and releasing instances, as EC2 was still experiencing provisioning failures. At 15:28, Mac M4 machines became temporarily unavailable because our connectivity to the Mac environment relies on AWS networking. As network stability improved, Mac workloads were able to resume. At 17:10, machine jobs \(Linux, Remote Docker, Mac M1 and M2\) began starting again after AWS applied a fix to EC2. At 17:30, Mac M4 machines began to successfully boot again after AWS applied fixes to the network load balancers. At the same time, CircleCI’s task queue hit its peak and began to decrease as AWS service health continued to improve. The backlog of jobs that had built up during the incident continued to clear over the next few hours. By 22:00, all customer jobs in the backlog had been processed, and new incoming jobs were being processed at pre-incident rates. We continued to monitor our systems, and the CircleCI incident was marked resolved at 22:15. ## Future Prevention and Process Improvement While outages of this scale are rare, we are strengthening both our resilience and the experience our customers have during major upstream disruptions. During this incident, we evaluated options to shift workloads to backup compute providers and different AWS regions. Based on the errors we were observing across multiple AWS services as well as instability in several upstream dependencies also impacted by the outage, we did not have confidence that shifting workloads would improve the situation. There was a significant risk of introducing new failures without reducing the disruption customers were already experiencing. In addition, based on the duration and scope of similar AWS incidents in the past, it was unlikely that shifting workloads mid-incident would restore service faster than AWS could complete their recovery. As a result, jobs continued to queue and then ran as expected once AWS restored stability. Additionally, in the event of an AWS incident anticipated to span multiple days or longer, we would initiate disaster recovery to a different region. However, in this particular case, we had information from AWS indicating that the service would be restored in a timeframe where waiting was the best approach for our customers. In order to ensure that our customers have the information they need to make timely decisions, we are also investing in improving our status page communication during major incidents. These improvements include: * Clearer visibility into which CircleCI functionality is impacted at any point in time. * More frequent updates based on our own service health, not only upstream reports. * Better expectations for how jobs will behave during disruptions. Our goal is to improve transparency, recovery options, and the predictability of the customer experience when the unexpected happens.

Read the full incident report →

Started Oct 10, 2025, 08:29 PM UTC· resolved Oct 10, 2025, 10:23 PM UTC

1. investigating Oct 10, 2025, 08:29 PM UTC

   Some customers may have experienced delays of around 5 minutes starting Docker Jobs.

2. investigating Oct 10, 2025, 09:04 PM UTC

   We are continuing to investigate the issue. Thank you for your patience.

3. identified Oct 10, 2025, 09:30 PM UTC

   The issue has been identified. We are rolling out the fix now. Wait times are returning to normal.

4. monitoring Oct 10, 2025, 10:17 PM UTC

   The issue impacting wait times for Docker jobs has been fixed and wait times are back to normal. We are currently monitoring our systems to ensure stability and confirm full resolution.

5. resolved Oct 10, 2025, 10:23 PM UTC

   This incident has been resolved. Wait times are back to normal. We thank you for your patience while our engineers fixed the underlying cause.

Read the full incident report →

Started Oct 10, 2025, 02:34 PM UTC· resolved Oct 10, 2025, 11:25 PM UTC

1. identified Oct 10, 2025, 02:34 PM UTC

   An issue has been identified causing a delay in available usage API data. Customers may experience issues retrieving usage reports for recent timeframes.

2. identified Oct 10, 2025, 05:36 PM UTC

   We traced the issue to a vendor and have reached out to them. As such, time to resolution remains unknown. Note: this does not affect historical data, just data for yesterday, October 9th.

3. monitoring Oct 10, 2025, 07:21 PM UTC

   Recovery in progress. Currently monitoring.

4. monitoring Oct 10, 2025, 09:05 PM UTC

   Recovery is ongoing. Apologies for the delay. We will provide updates as we get them. Timing remains tbd.

5. resolved Oct 10, 2025, 11:25 PM UTC

   The incident has been resolved. We appreciate your patience. We will continue to monitor throughout the weekend.

Read the full incident report →