Sauce Labs experienced a major incident on March 6, 2025 affecting US-West and US-West and 1 more component, lasting 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 06, 2025, 12:18 PM UTC
We are currently experiencing significantly reduced availability of Android Real Devices in all our datacenters. We are investigating.
- resolved Mar 06, 2025, 12:20 PM UTC
After taking remedial action, Android Real Devices are now available in all our datacenters. This incident is resolved
- postmortem Apr 01, 2025, 04:34 PM UTC
### **Dates:** Thursday March 6th 2025, 12:00 - 12:22 UTC ### **What happened:** Customers were unable to start new Android Real device tests in all datacentres. ### **Why it happened:** During a deployment, a race condition occurred during the reallocation of devices from "old" device pools to "new" device pools which caused devices to become unavailable in both pools. ### **How we fixed it:** The "old" pools were shutdown releasing the device lock, allowing the "new" pools to acquire the lock. ### **What we are doing to prevent it from happening again:** We are improving our device allocation processes.