Sauce Labs Outage History

### **Dates:** Wednesday, May 13th 2026, 18:47 UTC - Friday, May 15th 2026, 09:23 UTC. ### **What happened:** Customers were unable to access test run summaries for Real Device Cloud \(RDC\) jobs because events stopped publishing to the jobs Kafka topic in US-EAST. ### **Why it happened:** An authentication key used by the message producer unexpectedly lost its permissions during an account cleanup. ### **How we fixed it:** Manually restored the required permissions to re-establish the connection and resume service. ### **What we are doing to prevent it from happening again:** Migrating to a permanent service account, implementing a Dead Letter Queue \(DLQ\) for the jobs Kafka topic, and replaying the missing events to restore customer data.

### **Dates:** Thursday, April 23rd 2026, 22:43 UTC - Friday, April 24th 2026, 15:29 UTC ### **What happened:** Video assets were missing for virtual iOS simulator tests on ARM and macOS ARM desktop tests in the US-West and EU data centers. ### **Why it happened:** A product defect was introduced resulting in a screen capture failure. ### **How we fixed it:** We performed a rollback to a stable version. ### **What we are doing to prevent it from happening again:** We are improving monitoring & alerting to enhance our post deployment validation.

### **Dates:** Thursday, April 16th 2026, 00:00 UTC – 09:15 UTC ### **What happened:** Live and automated tests on iOS 17.0 simulators failed to start in both the EU and US-West data centers. Customers running tests on iOS 17.0 Intel-based simulators were unable to execute their tests for approximately 9 hours. ### **Why it happened:** A deployment introduced an incompatibility affecting iOS 17.0 on Intel-based infrastructure. The issue was not caught prior to release due to insufficient post-deployment test coverage for that specific simulator configuration. ### **How we fixed it:** We performed a rollback to the previous deployment, which restored full iOS 17.0 simulator functionality. ### **What we are doing to prevent it from happening again:** We are reviving and expanding automated post-deployment tests to cover a broader range of simulator configurations, including legacy Intel-based iOS versions, to catch incompatibilities before they reach production.

### **Dates:** Monday April 7th 2026, ~11:00 – 15:55 UTC ### **What happened:** Some customers experienced 503 errors when running tests via saucectl. The test-composer service was intermittently unavailable, preventing framework-based test execution. ### **Why it happened:** A stale Docker image was deployed to the test-composer service due to a packaging issue that arose during an internal container registry migration. This caused service pods to crash. ### **How we fixed it:** We identified the stale image and redeployed the correct version, restoring the service. ### **What we are doing to prevent it from happening again:** We are hardening our image deployment pipeline and adding validation checks to ensure container registry migrations do not result in stale or incorrect images being deployed to production.

### **Dates:** Tuesday, March 24th 2026, 09:32 UTC – 15:13 UTC ### **What happened:** Network calls failed on iOS devices during Real Device Cloud sessions where network capture was enabled. Approximately 12-13% of iOS sessions were affected. Android was not impacted. ### **Why it happened:** A deployment introduced a DNS resolution change that was incompatible with the iOS platform, causing network capture to break. ### **How we fixed it:** Rolled back the deployment to restore service. ### **What we are doing to prevent it from happening again:** Adding synthetic tests to catch network capture regressions before production, and implementing monitoring alerts for faster detection after deployments.

### **Dates:** Wednesday, March 19 2026, 04:45 UTC - 10:47 UTC. ### **What happened:** Approximately 15% of iOS devices in our US-West data center were temporarily unavailable for customer test sessions due to failed internet connectivity checks. ### **Why it happened:** An automated wireless network optimization feature adjusted transmit power levels on access points serving the affected devices, degrading wireless connectivity and causing devices to fail their availability checks. ### **How we fixed it:** The affected access points were identified and restarted, restoring normal wireless connectivity. ### **What we are doing to prevent it from happening again:** Evaluation of the automated optimization tools and a monitoring improvement.

### **Dates:** Friday, March 13th 2026, 14:43 UTC - 15:11 UTC. ### **What happened:** Real Devices \(iOS and Android\) availability gradually decreased across all data centers. ### **Why it happened:** A product defect was introduced resulting in a small subset of Real Devices \(~10%\) failing to maintain required connectivity. ### **How we fixed it:** Rollback to a stable version. ### **What we are doing to prevent it from happening again:** Improve monitoring & alerting, enhance post deployment validation.

### **Dates:** Tuesday March 10th 2026, 17:52 - 23:34 UTC ### **What happened:** The majority of iOS devices across all regions became unavailable. ### **Why it happened:** Apple's [ppq.apple.com](http://ppq.apple.com) app verification endpoint was down, causing internal device monitoring checks to fail, bringing devices offline. ### **How we fixed it:** We temporarily disabled these device monitoring checks. ### **What we are doing to prevent it from happening again:** Improved external monitoring to catch outages of apple’s [ppq.apple.com](http://ppq.apple.com) endpoint, loosened device monitoring to not take down live iOS devices if [ppq.apple.com](http://ppq.apple.com) is down.

### **Dates:** Friday, March 6th 2026, 21:38 UTC - 23:11 UTC ### **What happened:** During the incident timeline, customers running virtual iOS simulator tests on ARM or macOS ARM desktop tests in the EU Data Center were unable to start new sessions for either live or automated. ### **Why it happened:** There was a sequencing issue on the release of the ARM side disk images in the EU. ### **How we fixed it:** The image reference for the ARM side disk was rolled back to the previous reference to restore service. ### **What we are doing to prevent it from happening again:** The tests that run to validate the image syncing have been completed in each region.

### **Dates:** Friday, February 27th 2026, 16:15 UTC - 19:50 UTC ### **What happened:** Requests made using API client authentication would return 500 errors. ### **Why it happened:** An internal data structure became corrupted due to a race condition. ### **How we fixed it:** The affected service was restarted, and a long-term fix was applied. ### **What we are doing to prevent it from happening again:** Thread locking has been applied to the affected service.

### **Dates:** Monday, February 23rd 2026, 17:22 UTC - 22:31 UTC ### **What happened:** WDIO-based tests run by customers using Sauce Connect 4 could not be started. ### **Why it happened:** Misconfiguration caused failing health checks in some cases, causing customer tunnels to shut down. ### **How we fixed it:** Misconfiguration was corrected. ### **What we are doing to prevent it from happening again:** Evaluation of the underlying software stack. Customers who are still using SC4 that can migrate to SC5 should do so at their earliest convenience.

### **Dates:** Thursday, January 29th 2026, 19:30 UTC - 22:25 UTC. ### **What happened:** Application uploads and automated tests in the US-West-1 data center experienced intermittent errors between 5:43 pm \(UTC\) and 9:45 pm \(UTC\) ### **Why it happened:** Due to a high number of concurrent uploads and a temporary change to the network topology schema, the App Storage Service experienced increased connection latency and delays in acquiring connections to the backend. ### **How we fixed it:** App Storage Service network topology schema was restored to its original state. ### **What we are doing to prevent it from happening again:** We are improving monitoring, alerting and validations for the App Storage Service backend connectivity. We are working to implement processes to handle periods of increased load and proactively detect performance degradation to implement automated remediation and self-healing capabilities.

### **Dates:** Thursday, January 29th 2026, 07:04 - 08:46 UTC. ### **What happened:** Between 07:04 UTC and 08:46 UTC on January 29, 2026, customers using our US-West-1 region experienced instability with Mac resources. Specifically: * Approximately 20% of iOS simulator jobs failed to start due to infrastructure errors. * 10% of Virtual MacOS/iOS jobs failed to upload assets to S3. As a result, logs, videos, and screenshots for these specific jobs were lost and remain unavailable for download. ### **Why it happened:** Scheduled maintenance on a primary network circuit coincided with high traffic volume, which led to congestion on the backup circuit. ### **How we fixed it:** Full connectivity was restored when traffic was routed back to the primary circuit following the completion of maintenance. ### **What we are doing to prevent it from happening again:** Review network redundancy strategies and consider upgrading backup circuit capacity to ensure it can support peak traffic loads.

### **Dates:** Tuesday, January 13th 2025, 9:10 UTC - 10:37 UTC. ### **What happened:** The App Storage Service experienced high response times in the US-West-1 datacenter. ### **Why it happened:** Due to a high number of concurrent uploads, the App Storage Service experienced increased connection latency and delays in acquiring connections to the backend. ### **How we fixed it:** Service was restored by restarting the application. ### **What we are doing to prevent it from happening again:** We are improving monitoring, alerting and validations for the App Storage Service to backend connectivity. We are also evaluating how to handle periods of increased load and proactively detect performance degradation to implement automated remediation and self-healing capabilities.

### **Dates:** Monday, January 5th 2026, 20:40 UTC - 21:04 UTC. ### **What happened:** There was a decrease in Real Device availability in the US-West-1 datacenter. ### **Why it happened:** During a planned traffic flow migration, a network policy applied to the new traffic path was more restrictive than intended and this resulted in a subset of Real Devices \(~40%\) failing to maintain required connectivity. Although the change followed standard change management and peer review processes, the issue was not identified prior to activation. ### **How we fixed it:** The network policy was corrected and affected services recovered. ### **What we are doing to prevent it from happening again:** We are reviewing our network change processes to provide earlier detection of unintended behaviour during planned changes.

### **When it happened:** Wednesday, December 17th 2025, 10:37 UTC - Thursday, December 18th 2025, 11:59 UTC ### **What happened:** Applications were unable to be downloaded via Mobile App Distribution for a subset of customers. ### **Why it happened:** A product change combined with an increased and sustained load resulted in higher resource usage than anticipated, exceeding established limits for certain clients. ### **How we fixed it:** The service was rolled back to a stable version. ### **What we are doing to prevent it from happening again:** There are multiple improvements planned. 1. Improved monitoring & alerting, 2. Enhanced post-deployment validation 3. Capacity planning forecasting 4. More advanced communication where possible ahead of major changes 5. Adding canary to the roll out process

### **Dates:** Thursday, December 18th 2025, 5:32 UTC - 7:09 UTC. ### **What happened:** The App Storage Service experienced high response times in the US-West-1 datacenter. ### **Why it happened:** Due to a high number of concurrent uploads, the App Storage Service experienced increased connection latency and delays in acquiring connections to the backend. ### **How we fixed it:** The service was restored by restarting the application. ### **What we are doing to prevent it from happening again:** We are implementing improved monitoring, alerting and validations for App Storage Service backend connectivity. We will also look to enhance processes to handle periods of increased load and detect performance degradation proactively, in order to implement automated remediation and self-healing capabilities.

### **Dates:** Monday November 4 2025, 04:35 UTC - 08:20 UTC. ### **What happened:** The web UI for Insights and Test Results was intermittently unresponsive. ### **Why it happened:** A data store of components of the UI experienced unexpectedly high load, resulting in an unresponsive service. ### **How we fixed it:** The issue was resolved without human intervention. ### **What we are doing to prevent it from happening again:** We have improved support for the 3rd party vendor's logging and metrics tools.

### **Dates:** Monday November 3 2025, 12:40 UTC - 14:25 UTC. ### **What happened:** The Sauce Labs dashboard for Insights and Test Results was unresponsive, causing issues showing test results and related information. ### **Why it happened:** A data store for some components of the dashboard experienced unusually high load, resulting in an unresponsive service. ### **How we fixed it:** The issue was resolved by increasing the resources for the affected data store service. ### **What we are doing to prevent it from happening again:** Support for the 3rd party vendor's logging and metrics tools has been improved.

### **Dates:** Monday, October 27th 2025, 15:27 UTC - Wednesday, October 29th 2025, 11:34 UTC. ### **What happened:** Real Device Live Testing availability gradually decreased in the US-West-1 data center. ### **Why it happened:** A product defect was introduced which caused Live Testing sessions on Real Devices to fail to start. ### **How we fixed it:** We performed a rollback to the previous working release. ### **What we are doing to prevent it from happening again:** We have improved monitoring & alerting and enhanced our post-deployment validation processes.

### **Dates:** Thursday, October 9th 2025, 10:28 UTC - Tuesday, November 11th 2025, 11:34 UTC. ### **What happened:** Virtual Android testing availability gradually decreased in the US-West-1 data center, this led to increased error rates in starting sessions, The error rates remained high between November 4th and November 11th. ### **Why it happened:** An increased and sustained load on our service led to reduced availability of Android virtual VMs, increasing wait times to start sessions, which in turn led to an elevated number of failed sessions. ### **How we fixed it:** This was resolved by increasing infrastructure capacity and optimizing resource utilization. ### **What we are doing to prevent it from happening again:** We have improved monitoring & alerting, enhanced post-deployment validation and improved capacity planning.

### **Dates:** Tuesday, October 28th 2025, 14:03 UTC - 15:18 UTC \(primary incident\) Tuesday, October 28th 2025, 15:18 UTC - Thursday, October 30th 2025, 8:25 UTC \(~1% of devices impacted window\) ### **What happened:** Test executions using the 'stable' Appium version gradually decreased in the US-West-1, US-East-4, and EU-Central-1 data centers. ### **Why it happened:** A product defect was introduced which caused tests using the stable Appium version to fail immediately upon creation. ### **How we fixed it:** We preformed a rollback to a previous version on the majority of devices on Tuesday, October 28th 2025, 15:18 UTC, With a full service restoration across all devices completed on Thursday, October 30th 2025, 8:25 UTC. ### **What we are doing to prevent it from happening again:** We have improved monitoring & alerting and enhanced our post-deployment validation processes.

### **Dates:** Tuesday, October 28th 2025, 11:40 UTC - 14:05 UTC. ### **What happened:** Windows based tests in the EU data center were failing on startup which led to a decrease in availability of Windows-based Virtual Machines. ### **Why it happened:** A product defect was introduced which caused an issue with starting Windows-based VMs. ### **How we fixed it:** The deployment was rolled back to a stable version. ### **What we are doing to prevent it from happening again:** We have improved monitoring & alerting and enhanced our post-deployment validation.

### **Dates:** Wednesday, October 1st 2025, 10:34 UTC - Thursday, October 2nd 2025, 22:22 UTC ### **What happened:** Virtual Android and Visual tests using the “Android GoogleAPI Emulator” were failing if they contained conditioned that required the exact app dimensions. ### **Why it happened:** A new default emulator was released including a setting that did not account for customer defined virtual screen resolutions. ### **How we fixed it:** The issue was resolved by rolling back the deployment. ### **What we are doing to prevent it from happening again:** A new release was scheduled with an update to accommodate tests with customer defined screen resolution.