Orchestrator service is facing outage in Community Region

Update timeline

1. investigating May 11, 2026, 01:33 PM UTC

   We are investigating the cause of the outage for Orchestrator in Community Region and are working on a fix.

2. monitoring May 11, 2026, 01:39 PM UTC

   A fix has been deployed for Orchestrator in Community region and we are monitoring closely. Next update: We are continuing to monitor system behavior.

3. resolved May 11, 2026, 01:43 PM UTC

   This incident has been resolved and service stability has been confirmed for Orchestrator. Impact: All impacted features are operating normally.

4. postmortem May 26, 2026, 07:37 AM UTC

   ## Customer Impact Between May 11, 2026 at 13:00 UTC and 13:35 UTC, Automation Cloud subset of customers experienced widespread errors and significant slowdowns across Orchestrator. During this 35-minute window, starting and managing automation jobs failed or timed out, and operations that did not directly involve jobs were also degraded as the impact propagated to the rest of the product. Enterprise tenants were not affected. ## Root Cause The incident was triggered by a routine product deployment to Community Orchestrator that included a schema change to an existing column on a core Orchestrator database table — specifically, a change to the column's size. Based on prior changes of a similar shape, the team expected this to be applied as a fast, metadata-only operation with no service impact. In practice, the database applied the change as a heavier operation that required an exclusive lock on that table for the duration of its execution, preventing Orchestrator from reading from or writing to the affected table while the lock was held. Because most parts of Orchestrator share the same database, the saturation caused by the blocked operation propagated to the rest of the product, not only to functionality that directly used the affected table. The lock — and the resulting customer impact — was released when the operation timed out automatically after approximately 35 minutes. ## Detection Automated monitoring detected the issue within minutes of the deployment being applied. A critical alert was fired when database worker utilization on the Community Orchestrator database exceeded its threshold, and our engineering team was paged and acknowledged within minutes. Elevated error rates and latency across multiple Orchestrator APIs were confirmed shortly after through additional service-level alerts. ## Response On-call engineers correlated the saturation alert with the recently applied database change and identified table-level contention as the source of the impact. The team prepared mitigation options, including manually terminating the blocking operation, but service was restored automatically at 13:35 UTC when the operation timed out and released the lock. After recovery, the team verified that connection-pool utilization, throughput, and error rates across Orchestrator had returned to normal levels, and that no in-flight customer operations had been left in an inconsistent state. ## Follow-Up To prevent recurrence, we are implementing the following improvements: * **Stricter review of schema changes before deployment**, including explicit pre-flight verification of whether a given column or table change will actually be applied as a fast metadata-only operation on the target table at its current size and database version — with mandatory sign-off when it will not. * **Improved runbook for database-contention incidents triggered by deployments**, so that on-call engineers can quickly identify a deployment-introduced blocking operation and safely terminate it rather than waiting for an automatic timeout.