Degraded Performance for Orchestrator in US region

Update timeline

1. resolved Jun 25, 2026, 02:46 PM UTC

   The Orchestrator service in the East US region experienced increased latency and a subset of failed requests between June 25, 14:06 UTC and June 25, 14:12 UTC. During this period, customers may have encountered slower response times and intermittent request failures. The issue has been mitigated, and the service is now operating normally. We are continuing to review the incident and publish the root cause. Affected Services: East US – Orchestrator

2. postmortem Jul 30, 2026, 07:34 AM UTC

   ## Customer Impact Between June 25, 2026 at 2:06 pm UTC and June 25, 2026 at 2:12 pm UTC, Orchestrator in the US region experienced increased latency and a subset of failed requests. During this six-minute window, customers may have seen slower Orchestrator response times and intermittent request failures. The impact was limited to customers using Orchestrator in the US region. Existing connections and overall database availability were not affected, and no data was lost. ## Root Cause The disruption was caused by an internal maintenance operation, which made a brief configuration change to the database - a change that momentarily acquires a lock normally released in a predictable short time. On this occasion the operation was interrupted before completing, so the lock was not released on schedule and stayed in place for several minutes until the database's automatic cleanup released it. While the lock was held, the database remained online and existing connections continued to work; only the establishing of new database connections was blocked, which produced the increased latency and failed requests during the window. ## Detection The issue was identified through automated monitoring for Orchestrator in the US region, which began alerting within approximately two minutes of the impact. Our formal incident response process was initiated on June 25, 2026 at 2:24 pm UTC; by that point the customer-impacting window had already ended and the service had recovered. ## Response At 2:24 pm UTC on June 25, 2026, our incident response process was activated and responders were engaged. Initial investigation examined the network and API management layers and considered a possible platform-level database event; both were ruled out as the cause. At 2:33 pm UTC, service monitoring confirmed the customer-impacting window as 2:06 pm UTC to 2:12 pm UTC in the US region. A status page update was requested at 2:33 pm UTC and published at 2:46 pm UTC with the confirmed impact window and affected service. Because the service had already returned to normal operation at 2:12 pm UTC, the incident was marked resolved at 2:46 pm UTC. The confirmed root cause was established subsequently through detailed investigation with our cloud database provider. ## Follow Up Root cause has been confirmed and we are taking the following actions to prevent recurrence and reduce the impact of similar events: * Hardening the internal maintenance tooling so an interrupted operation always releases its resources cleanly, even if stopped midway. * Tightening the controls governing how and when such maintenance operations run against production. * Improving monitoring to detect this lock/connection condition earlier.