Sauce Labs incident

2025-July-2 Service Incident

Critical Resolved View vendor source →

Sauce Labs experienced a critical incident on July 2, 2025 affecting EU-Central and EU-Central and 1 more component, lasting 9h 59m. The incident has been resolved; the full update timeline is below.

Started
Jul 02, 2025, 10:11 AM UTC
Resolved
Jul 02, 2025, 08:11 PM UTC
Duration
9h 59m
Detected by Pingoru
Jul 02, 2025, 10:11 AM UTC

Affected components

EU-CentralEU-CentralEU-Central

Update timeline

  1. investigating Jul 02, 2025, 10:11 AM UTC

    We are currently experiencing reduced availability of Real Devices in our EU-Central-1 datacenter. We are investigating.

  2. investigating Jul 02, 2025, 11:12 AM UTC

    We are continuing to see reduced availability of Real Devices in our EU-Central-1 datacenter. We are continuing to investigate.

  3. investigating Jul 02, 2025, 11:50 AM UTC

    We have identified a networking issue in the EU-Central-1 datacenter, resulting in a high error rate in EU Real Device tests. We are continuing to investigate.

  4. investigating Jul 02, 2025, 04:19 PM UTC

    We are actively working on a solution to resolve the networking issues being seen in the EU datacenter, which involves replacing failed hardware. A high error rate will continue to be seen with EU Real Device tests in the meantime.

  5. resolved Jul 02, 2025, 08:11 PM UTC

    Our Android devices are now fully operational in EU-Central Data Center. We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center, which is leading to degraded availability for our iOS user base. We have identified the root cause and implementing a final fix. We expect to restore full availability for all iOS users by tomorrow, Thursday, July 3, 2025, end of day

  6. postmortem Jul 16, 2025, 10:45 AM UTC

    ### **Dates:** Wednesday July 2nd 2025, 09:74 UTC - 17:19 UTC. ### **What happened:** Approximately half of our real devices in the EU-Central-1 datacenter became unavailable for testing. ### **Why it happened:** There was a critical failure with a device within our network infrastructure. ### **How we fixed it:** The issue was resolved by replacing the failed device. ### **What we are doing to prevent it from happening again:** We are reviewing the current network infrastructure strategy to improve resiliency.