Sauce Labs experienced a major incident on October 30, 2024 affecting US-West and US-West and 1 more component, lasting 2h 52m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 30, 2024, 07:00 PM UTC
We are currently seeing intermittent issues with Sauce Connect tunnel startup and allocation in the US-West-1 data center. We are investigating.
- monitoring Oct 30, 2024, 08:53 PM UTC
We have taken remedial action and are seeing an improvement with Sauce Connect tunnel startup and allocation in the US-West-1 data center. We are monitoring
- resolved Oct 30, 2024, 09:52 PM UTC
After taking remedial action all services are operating as normal. This incident is resolved.
- postmortem Apr 01, 2025, 03:39 PM UTC
### **Dates:** Wednesday October 30th 2023, 03:45 - 21:06 UTC ### **What happened:** Intermittent errors occurred when starting or using Sauce Connect tunnel in the US West datacenter. ### **Why it happened:** A service responsible for creating new bindings between the tunnel endpoints and test VMs experienced timeouts on some hosts. ### **How we fixed it:** Service was restored after clearing bindings on the affected hosts. ### **What we are doing to prevent it from happening again:** Monitoring and alerting for when the service is unable to create new bindings has been improved. The root cause of the condition is under investigation.