Sauce Labs incident

2024-November-19 Service Incident

Major Resolved View vendor source →

Sauce Labs experienced a major incident on November 19, 2024 affecting US-West and US-West and 1 more component, lasting 53m. The incident has been resolved; the full update timeline is below.

Started
Nov 19, 2024, 09:35 PM UTC
Resolved
Nov 19, 2024, 10:28 PM UTC
Duration
53m
Detected by Pingoru
Nov 19, 2024, 09:35 PM UTC

Affected components

US-WestUS-WestUS-WestEU-CentralEU-CentralEU-CentralUS-EastUS-EastUS-East

Update timeline

  1. investigating Nov 19, 2024, 09:35 PM UTC

    Android real devices in US West 1, EU Central 1, and US East datacenter are unable to run automated Appium and Espresso app tests and manual app tests. We are investigating.

  2. resolved Nov 19, 2024, 10:28 PM UTC

    After taking remedial action all services are operating as normal. This incident is resolved.

  3. postmortem Apr 01, 2025, 03:54 PM UTC

    ### **Dates:** Wednesday November 19th 2024, 21:30 - 23:30 UTC ### **What happened:** Android real devices were unable to run app tests in all datacenters. ### **Why it happened:** An unexpected policy change by Google caused MDM managed devices to be locked down due to policy violations around accessibility services. ### **How we fixed it:** We rolled back enablement of the TalkBack accessibility service. ### **What we are doing to prevent it from happening again:** We opened a support case with Google to get further information on why this policy change happened, as well as began investigating running our own MDM solution.