Sauce Labs Outage History

### **Dates:** Monday April 7th 2026, ~11:00 – 15:55 UTC ### **What happened:** Some customers experienced 503 errors when running tests via saucectl. The test-composer service was intermittently unavailable, preventing framework-based test execution. ### **Why it happened:** A stale Docker image was deployed to the test-composer service due to a packaging issue that arose during an internal container registry migration. This caused service pods to crash. ### **How we fixed it:** We identified the stale image and redeployed the correct version, restoring the service. ### **What we are doing to prevent it from happening again:** We are hardening our image deployment pipeline and adding validation checks to ensure container registry migrations do not result in stale or incorrect images being deployed to production.

### **Dates:** Tuesday, March 24th 2026, 09:32 UTC – 15:13 UTC ### **What happened:** Network calls failed on iOS devices during Real Device Cloud sessions where network capture was enabled. Approximately 12-13% of iOS sessions were affected. Android was not impacted. ### **Why it happened:** A deployment introduced a DNS resolution change that was incompatible with the iOS platform, causing network capture to break. ### **How we fixed it:** Rolled back the deployment to restore service. ### **What we are doing to prevent it from happening again:** Adding synthetic tests to catch network capture regressions before production, and implementing monitoring alerts for faster detection after deployments.

### **Dates:** Wednesday, March 19 2026, 04:45 UTC - 10:47 UTC. ### **What happened:** Approximately 15% of iOS devices in our US-West data center were temporarily unavailable for customer test sessions due to failed internet connectivity checks. ### **Why it happened:** An automated wireless network optimization feature adjusted transmit power levels on access points serving the affected devices, degrading wireless connectivity and causing devices to fail their availability checks. ### **How we fixed it:** The affected access points were identified and restarted, restoring normal wireless connectivity. ### **What we are doing to prevent it from happening again:** Evaluation of the automated optimization tools and a monitoring improvement.

### **Dates:** Friday, March 13th 2026, 14:43 UTC - 15:11 UTC. ### **What happened:** Real Devices \(iOS and Android\) availability gradually decreased across all data centers. ### **Why it happened:** A product defect was introduced resulting in a small subset of Real Devices \(~10%\) failing to maintain required connectivity. ### **How we fixed it:** Rollback to a stable version. ### **What we are doing to prevent it from happening again:** Improve monitoring & alerting, enhance post deployment validation.

### **Dates:** Tuesday March 10th 2026, 17:52 - 23:34 UTC ### **What happened:** The majority of iOS devices across all regions became unavailable. ### **Why it happened:** Apple's [ppq.apple.com](http://ppq.apple.com) app verification endpoint was down, causing internal device monitoring checks to fail, bringing devices offline. ### **How we fixed it:** We temporarily disabled these device monitoring checks. ### **What we are doing to prevent it from happening again:** Improved external monitoring to catch outages of apple’s [ppq.apple.com](http://ppq.apple.com) endpoint, loosened device monitoring to not take down live iOS devices if [ppq.apple.com](http://ppq.apple.com) is down.

### **Dates:** Friday, March 6th 2026, 21:38 UTC - 23:11 UTC ### **What happened:** During the incident timeline, customers running virtual iOS simulator tests on ARM or macOS ARM desktop tests in the EU Data Center were unable to start new sessions for either live or automated. ### **Why it happened:** There was a sequencing issue on the release of the ARM side disk images in the EU. ### **How we fixed it:** The image reference for the ARM side disk was rolled back to the previous reference to restore service. ### **What we are doing to prevent it from happening again:** The tests that run to validate the image syncing have been completed in each region.

### **Dates:** Friday, February 27th 2026, 16:15 UTC - 19:50 UTC ### **What happened:** Requests made using API client authentication would return 500 errors. ### **Why it happened:** An internal data structure became corrupted due to a race condition. ### **How we fixed it:** The affected service was restarted, and a long-term fix was applied. ### **What we are doing to prevent it from happening again:** Thread locking has been applied to the affected service.

### **Dates:** Monday, February 23rd 2026, 17:22 UTC - 22:31 UTC ### **What happened:** WDIO-based tests run by customers using Sauce Connect 4 could not be started. ### **Why it happened:** Misconfiguration caused failing health checks in some cases, causing customer tunnels to shut down. ### **How we fixed it:** Misconfiguration was corrected. ### **What we are doing to prevent it from happening again:** Evaluation of the underlying software stack. Customers who are still using SC4 that can migrate to SC5 should do so at their earliest convenience.