Multiple Services degraded in US Region

Update timeline

1. resolved Apr 20, 2026, 12:52 PM UTC

   Between 12:02 and 12:14 UTC, an issue affecting one of our core services, Orchestrator, caused disruption across multiple services. The incident was triggered by unexpected restarts in the underlying infrastructure supporting Orchestrator. We understand the impact this has had on our customers and apologize for the disruption. We will publish a detailed RCA, including action items to prevent a recurrence.

2. postmortem Apr 24, 2026, 11:38 AM UTC

   ## Customer Impact Between 12:02 pm UTC and 12:15 pm UTC on April 20, 2026, customers in the U.S. region experienced elevated error rates and failed requests when using the Orchestrator service. During this approximately 13-minute window, automation workflows were interrupted and a portion of requests returned errors. Error rates began declining by 12:14 pm UTC and returned to normal by 12:15 pm UTC. Impact was limited to the Orchestrator service in the U.S. region. All other services and regions remained fully operational. ## Root Cause A routine infrastructure upgrade in the U.S. region restarted all servers supporting the Orchestrator service within a four-minute window—far more rapidly than intended. A safeguard designed to limit how many service instances can be unavailable at once was in place, but was not honored during the upgrade, so nearly all Orchestrator instances went offline simultaneously. Remaining capacity was insufficient to serve normal traffic, resulting in widespread errors until instances came back online. The exact mechanism by which the upgrade bypassed the safeguard is under deeper investigation. ## Detection Automated monitoring detected a sudden increase in errors at 12:05 pm UTC—approximately three minutes after the impact began. Engineers acknowledged the alerts and began investigating immediately. Customer reports arrived within the same window and corroborated the alerts. ## Response Engineers assembled on an incident call and quickly identified that all Orchestrator service instances had restarted nearly simultaneously, correlating with an ongoing infrastructure upgrade. Underlying database and service dependencies were verified as healthy, focusing the investigation on the upgrade process itself. The service recovered on its own as instances came back online. **April 20, 2026** * 12:12 pm UTC— Incident call assembled— ongoing upgrade identified as the cause. * 12:14 pm UTC— Error rates dropped below 2% as instances recovered. * 12:15 pm UTC— Service returned to normal operation. * 12:26 pm UTC— Full-service health confirmed—Remaining Orchestrator upgrades paused. * 12:52 pm UTC— Status page updated and the incident was marked resolved shortly after. ## Follow-Up We are implementing improvements to reduce the likelihood of similar incidents. _Short-term improvements_ * Investigate the exact mechanism by which the upgrade bypassed the restart safeguard, and add automated validation to enforce it during all maintenance activities. * Add pre-upgrade verification to confirm service capacity will be maintained before any changes are applied. * Add monitoring and alerts for unexpected patterns of simultaneous service restarts. _Long-term improvements_ * Update operational procedures and team training to reinforce best practices for safe infrastructure upgrades.