Sauce Labs Outage History

### **Dates:** Wednesday September 5th 2025, 13:50 UTC - 16:31 UTC ### **What happened:** The App Storage Service in the EU Central 1 data center experienced elevated response times, leading to timeout errors with uploading applications. ### **Why it happened:** The App Storage Service experienced connection issues with the backend. ### **How we fixed it:** The connection issues were resolved by restarting the affected application. ### **What we are doing to prevent it from happening again:** We have improved observability signals for the App Storage Service and implemented regular scheduled restarts of the application.

### **Dates:** Friday August 19th 2025, 16:20 UTC - 16:55 UTC. ### **What happened:** During the times stated, automated Android real device tests were failing to start intermittently in the US-West-1 datacenter. ### **Why it happened:** A defect was introduced by a deployment which impacted Appium tests. ### **How we fixed it:** The deployment was rolled back. ### **What we are doing to prevent it from happening again:** We have improved our internal test cases to detect the conditions in which the defect occurred.

### **Dates:** Tuesday July 22d 2025, 07:28 UTC - 15:07 UTC. ### **What happened:** The Sauce Labs dashboard for our US-West-1 datacenter was unavailable intermittently during the incident timeline. ### **Why it happened:** This issue was caused by Cache poisoning via a specific header. ### **How we fixed it:** The header settings were reviewed and fixed in our proxies. ### **What we are doing to prevent it from happening again:** We have added stricter rules to prevent further cache poisoning attempts.

### **Dates:** Wednesday July 2nd 2025, 09:74 UTC - 17:19 UTC. ### **What happened:** Approximately half of our real devices in the EU-Central-1 datacenter became unavailable for testing. ### **Why it happened:** There was a critical failure with a device within our network infrastructure. ### **How we fixed it:** The issue was resolved by replacing the failed device. ### **What we are doing to prevent it from happening again:** We are reviewing the current network infrastructure strategy to improve resiliency.

### **Dates:** Wednesday June 18th 2025, 11:04 UTC - 12:23 UTC ### **What happened:** The service responsible for App storage and retrieval experienced long response times. ### **Why it happened:** The App storage service was experiencing connectivity issues. ### **How we fixed it:** The connectivity issues disappeared before any remediation actions could be taken.. ### **What we are doing to prevent it from happening again:** We have added tracing validations for the app storage service connectivity to give earlier visibility of any future occurrences.

### **Dates:** Monday June 16th 2025, 12:25 UTC - 13:20 UTC ### **What happened:** Approximately 50% of RDC devices in the EU-Central-1 datacenter were unavailable. ### **Why it happened:** Maintenance on the datacenter power supply led to devices being shut down. ### **How we fixed it:** The affected devices were restarted. ### **What we are doing to prevent it from happening again:** We are looking to add more power redundancy and create established maintenance windows.

### **Dates:** Friday June 13th 2025, 07:47 UTC - 09:07 UTC ### **What happened:** A release impacted components of the Sauce Labs Web UI, making them unusable. ### **Why it happened:** Certain fonts were blocked from loading due to CORS issue. ### **How we fixed it:** The release was rolled back. ### **What we are doing to prevent it from happening again:** We are adding additional validation steps to prevent this from recurring.

### **Dates:** Thursday May 13th 2025, 10:26 UTC - Friday May 14th 2025 16:48 UTC ### **What happened:** Video recordings for Android tests intermittently failed in the US-West-1 datacenter. ### **Why it happened:** A service involved in capturing recordings entered a deadlocked state when receiving a specific combination of requests and options. ### **How we fixed it:** We cleared the deadlocked service. ### **What we are doing to prevent it from happening again:** We are introducing additional monitors and automatic recovery for the deadlocked condition.

### **Dates:** Wednesday May 14th 2025, 08:00 UTC - 09:03 UTC ### **What happened:** An authentication failure lead to Real Device live & automated app tests our in US-East-4 Datacenter. ### **Why it happened:** Connections from the App Storage service failed due to a database issue. ### **How we fixed it:** The database cluster required restarting. ### **What we are doing to prevent it from happening again:** Database clean-ups will be done only during maintenance windows.

### **Dates:** Tuesday April 6th 2025, 03:38 UTC - 03:58 UTC ### **What happened:** RDC devices were unavailable in the EU-Central-1 Datacenter. ### **Why it happened:** There was a DNS caching failure during a third party provider’s maintenance. ### **How we fixed it:** Availability was restored automatically, after the instances completed their start-up. ### **What we are doing to prevent it from happening again:** We will update our DNS caching for better fault tolerance.

### **Dates:** Wednesday April 23rd 2025, 16:50 UTC - 17:50 UTC ### **What happened:** Native Android tests failed in the EU-Central-1 datacenter. ### **Why it happened:** A large increase in simultaneous jobs with unique applications and cache misses, exhausted resources on the hosts responsible for installing Android native applications. ### **How we fixed it:** Resources were increased to cope with demand. ### **What we are doing to prevent it from happening again:** We have introduced additional controls on native application installs to prevent simultaneous downloads of the same app to a single host .

### **Dates:** Tuesday April 1st 2025, 14:54 UTC - Wednesday April 2nd 2025, 13:45 UTC ### **What happened:** iOS tests using Sauce Connect failed to start. ### **Why it happened:** There was a mismatch in the SSL certificate's expiration dates. ### **How we fixed it:** Re-ordered new SSL certificates. ### **What we are doing to prevent it from happening again:** We are adding monitoring for the SSL certificates.

### **Dates:** Wednesday March 26th 2025, 19:15 - Thursday March 27th 2025, 14:30 UTC ### **What happened:** An internal cache for mobile applications in one datacenter was unable to reach its origin, causing artifacts to age out and no longer be available to be served. ### **Why it happened:** A change to the cache’s routing prevented it from reaching its origin. ### **How we fixed it:** The routing change was reverted. ### **What we are doing to prevent it from happening again:** Additional monitoring and alerting has been implemented to identify this issue more quickly in the future.

### **Dates:** Thursday March 6th 2025, 12:00 - 12:22 UTC ### **What happened:** Customers were unable to start new Android Real device tests in all datacentres. ### **Why it happened:** During a deployment, a race condition occurred during the reallocation of devices from "old" device pools to "new" device pools which caused devices to become unavailable in both pools. ### **How we fixed it:** The "old" pools were shutdown releasing the device lock, allowing the "new" pools to acquire the lock. ### **What we are doing to prevent it from happening again:** We are improving our device allocation processes.

### **Dates:** Saturday February 22nd 2025, 00:00 - 04:46 UTC ### **What happened:** Customers were unable to create SauceConnect tunnels in our US-West-1 and EU-Central-1 regions. ### **Why it happened:** The SSL certificate for the Sauce Connect frontend had expired. ### **How we fixed it:** The certificate was renewed and deployed to the affected regions. ### **What we are doing to prevent it from happening again:** We are improving our alerting around SSL certificate expiration.

### **Dates:** Wednesday February 17th 2023, 12:52 - 13:53 UTC **What happened:** Real Device tests in EU-Central-1 datacenter were experiencing issues reaching internet destinations. ### **Why it happened:** A new internet provider was introduced that was experiencing issues with their network.. ### **How we fixed it:** A rollback was executed to move traffic off of the affected internet provider. ### **What we are doing to prevent it from happening again:** We engaged the provider and they corrected configuration issues with their network.

### Dates: Tuesday January 21st 2025, 12:40 - 13:44 UTC ### **What happened:** Tests using iOS Real Devices experienced failures to download and install apps for the eu-central01 and us-west-1 datacenters. **Why it happened:** A misconfiguration with the resigner service caused errors when communicating with the app storage. ### **How we fixed it:** A rollback was executed, restoring configuration to the previous working state. ### **What we are doing to prevent it from happening again:** * Improve canary deployments for each Real Device Cloud region. * Improve end to end testing for canary deployments. * Improve SLOs for the Real Device application installs

### **Dates:** Wednesday 8 January 2025, 11:00 - 12:25 UTC ### **What happened:** We were experiencing intermittent issues with Virtual device tests missing test assets in the US-West-1 region ### **Why it happened:** The asset uploader service was experiencing Http errors uploading assets to our backend storage. ### **How we fixed it:** The 3rd-party provider resolved an issue with their infrastructure. ### **What we are doing to prevent it from happening again:** We are improving the caching and retry logic of the asset uploader service to prevent further occurrence.

### **Dates:** Friday 3 January 2025, 09:52 - 11:07 UTC ### **What happened:** Customers were unable to access test results in the Web UI for our US-West-1 datacenter. ### **Why it happened:** A defect was introduced during a product deployment. ### **How we fixed it:** A rollback was executed to the previous working version. ### **What we are doing to prevent it from happening again:** We are creating additional checks for the authentication method upgrades.

### **Dates:** Wednesday December 18th 2024, 09:05 - 11:31 UTC ### **What happened:** Customers were unable to purchase self-service plans through the billing page. ### **Why it happened:** Breaking changes were introduced between our billing service client SDK and the third-party billing provider. ### **How we fixed it:** The SDK for the third-party billing provider was updated in our service. ### **What we are doing to prevent it from happening again:** We’re working with the third-party billing provider to better understand their versioning practices to ensure we remain inline with them.

### **Dates:** Thursday December 12 2024, 12:40 - 13:30 UTC ### **What happened:** Video recordings for Android Real Device tests were not showing in test results. ### **Why it happened:** A code deployment contained a broken dependency. ### **How we fixed it:** A rollback was executed to the previous known working version. ### **What we are doing to prevent it from happening again:** A check for broken dependencies was added to the deployment.

### **Dates:** Wednesday November 19th 2024, 21:30 - 23:30 UTC ### **What happened:** Android real devices were unable to run app tests in all datacenters. ### **Why it happened:** An unexpected policy change by Google caused MDM managed devices to be locked down due to policy violations around accessibility services. ### **How we fixed it:** We rolled back enablement of the TalkBack accessibility service. ### **What we are doing to prevent it from happening again:** We opened a support case with Google to get further information on why this policy change happened, as well as began investigating running our own MDM solution.

### **Dates:** Wednesday October 30th 2023, 03:45 - 21:06 UTC ### **What happened:** Intermittent errors occurred when starting or using Sauce Connect tunnel in the US West datacenter. ### **Why it happened:** A service responsible for creating new bindings between the tunnel endpoints and test VMs experienced timeouts on some hosts. ### **How we fixed it:** Service was restored after clearing bindings on the affected hosts. ### **What we are doing to prevent it from happening again:** Monitoring and alerting for when the service is unable to create new bindings has been improved. The root cause of the condition is under investigation.