2026-September-28 Service Incident

Update timeline

1. investigating Sep 28, 2026, 04:04 PM UTC

   We are currently seeing elevated error rates for automated real device tests in US-West-1 data center. We are currently investigating.

2. monitoring Sep 28, 2026, 04:56 PM UTC

   The root cause of the elevated error rates for Real Device tests in the US-West-1 data center has been identified. Remediation has been applied, and we are actively monitoring the situation.

3. resolved Sep 28, 2026, 05:19 PM UTC

   After taking remedial action, Real Device test error rates have returned to normal in the US-West-1 data center. All services are fully operational.

4. postmortem Oct 02, 2026, 07:15 PM UTC

   ### **Dates:** Monday, September 28th 2026, 14:19 UTC - 16:52 UTC. ### **What happened:** Real device tests in US West hung and failed to output test artifacts due to slow uploads. ### **Why it happened:** A degraded interconnect link suffered high packet loss from optical errors, but BGP stayed up, preventing automatic failover to healthy paths. ### **How we fixed it:** Manually disabled the degraded port on the edge router, forcing traffic onto healthy alternate interconnects. ### **What we are doing to prevent it from happening again:** Automating link draining based on CRC error thresholds, improving protocols for isolating flapping vendor links, and hardening upload resilience.