Degraded performance for Agent/Maestro LLM workflow runs in US region

Update timeline

1. investigating Aug 04, 2026, 07:14 PM UTC

   We are investigating reports of degraded performance impacting Agent/Maestro workflow runs that make LLM calls for Agents in the US region. Impact: Users may experience delayed or failed Agent/Maestro workflow runs when an LLM step is executed. Our teams are working to identify the cause and will share more details as the investigation progresses.

2. identified Aug 04, 2026, 07:19 PM UTC

   We have identified the issue causing degraded performance for Agent/Maestro workflow runs that make LLM calls for Agents in the US region. Impact: Users may continue to experience delayed or failed Agent/Maestro workflow runs when an LLM step is executed. Our teams are working on a fix and will share more details as remediation progresses.

3. monitoring Aug 04, 2026, 08:31 PM UTC

   We have implemented a fix for the issue causing degraded performance for Agent/Maestro workflow runs that make LLM calls for Agents in the US region. Impact: Users may have experienced delayed or failed Agent/Maestro workflow runs when an LLM step was executed. Our teams are monitoring the service to confirm continued stability and will share more details as needed.

4. monitoring Aug 04, 2026, 09:39 PM UTC

   We have implemented a fix for the issue causing degraded performance for Agent/Maestro workflow runs that make LLM calls for Agents in the US region. Impact: Users may have experienced delayed or failed Agent/Maestro workflow runs when an LLM step was executed. Our teams are continuing to monitor the fix for stability and will share more details as needed.

5. monitoring Aug 04, 2026, 10:53 PM UTC

   We have implemented a fix for the issue causing degraded performance for Agent/Maestro workflow runs that make LLM calls for Agents in the US region. Impact: Users may have experienced delayed or failed Agent/Maestro workflow runs when an LLM step was executed. Our teams are continuing to monitor the fix and confirm there is no further impact.

6. investigating Aug 05, 2026, 01:01 AM UTC

   We are continuing to investigate reports of degraded performance impacting Agent/Maestro workflow runs that make LLM calls for Agents in the US region. Impact: Users may experience delayed or failed Agent/Maestro workflow runs when an LLM step is executed. After applying mitigation steps and monitoring the service, we are still observing latency in some workflows. Our teams are investigating whether an upstream provider may be contributing to the continued latency and will share more details as the investigation progresses.

7. identified Aug 05, 2026, 02:10 AM UTC

   We have identified that the degraded performance impacting Agent/Maestro workflow runs that make LLM calls for Agents in the US region may be related to a third-party upstream dependency. Impact: Users may experience delayed or failed Agent/Maestro workflow runs when an LLM step is executed. Our teams are continuing to work with the upstream provider and will share more details as remediation progresses.

8. resolved Aug 05, 2026, 03:39 AM UTC

   The issue has been resolved and Agent/Maestro LLM workflow performance has returned to expected levels Impact: No ongoing user impact.

9. postmortem Aug 20, 2026, 05:46 PM UTC

   ## Customer impact Between August 4, 2026, at 3:25 AM UTC and August 5, 2026, at 3:39 AM UTC, a subset of customers in the US region experienced degraded performance affecting Agent and Maestro workflow runs. This issue was isolated to workflow steps executing Large Language Model (LLM) calls in tandem with PII masking. The event lasted approximately 24 hours. Affected users experienced delayed or failed executions, characterized by extended runtimes, workflows exceeding the five-minute threshold, and timeout errors. ## Root cause An upstream, third-party service powering our PII detection capabilities experienced a localized configuration anomaly within its US region. This restriction prevented the service from scaling dynamically to accommodate transaction volumes. The upstream provider verified the issue and applied a manual correction. The infrastructure constraint had caused requests involving PII detection and related LLM workflow steps to experience elevated latency or time out. ## Detection Initial support tickets regarding timeouts were received on August 4, 2026, at 3:25 AM UTC. Because failures were intermittent and a portion of requests continued to process successfully, an incident was not declared at that time. Following subsequent reports from multiple customers, a formal incident was declared at 6:27 PM UTC. Automated latency alerts did not trigger as intended, preventing proactive internal detection prior to customer escalation. ## Response At 7:14 PM UTC on August 4, 2026, we published a public status update acknowledging the degraded Agent and Maestro workflow performance in the US region. By 7:19 PM UTC, teams identified a probable cause and initiated remediation efforts. An initial mitigation patch was deployed at 8:31 PM UTC, followed by close performance monitoring; however, customer feedback at 8:41 PM UTC and 8:50 PM UTC indicated persistent latency. Continued investigation at 11:04 PM UTC confirmed ongoing elevated response times and timeout errors originating from the third-party service, prompting deeper escalation and joint investigation with the upstream cloud provider. At 2:06 AM UTC on August 5, 2026, the provider confirmed their scaling configuration issue was resolved. Telemetry indicated a downward trend in workflow durations by 2:22 AM UTC, with performance steadily normalizing over the subsequent three hours. The incident was officially closed at 3:39 AM UTC after Agent and Maestro metrics fully returned to baseline operational levels. ## Follow up - We are auditing our alerting thresholds to ensure subtle latency increases are captured and escalated earlier, including verification that throttling-related notifications route directly to on-call response teams. - We are evaluating high-capacity and automated scaling alternatives for the third-party service integration to mitigate future upstream infrastructure constraints. - We are refining our request management and rate-limiting policies to prevent disproportionate localized utilization from affecting concurrent workflows.