Degraded-performance in MLS Observer thruough DU - US

Update timeline

1. investigating Sep 03, 2026, 03:34 PM UTC

   We are currently investigaing the issue

2. identified Sep 03, 2026, 03:39 PM UTC

   The issue was identified, and the appropriate mitigation measures were implemented to address it.

3. monitoring Sep 03, 2026, 03:47 PM UTC

   The issue has been successfully resolved, and the service has been fully restored. The service is operating normally at this time.

4. resolved Sep 03, 2026, 04:17 PM UTC

   The issue has been successfully resolved, and the service has been fully restored. The service is operating normally at this time.

5. postmortem Sep 25, 2026, 11:11 AM UTC

   ## Customer impact Between approximately 3:20 pm UTC and 3:40 pm UTC on September 3, 2026, some customers using Document Understanding in the US experienced failed document processing requests. Affected requests returned service unavailable errors, and some extraction and processing calls did not complete successfully. The impact lasted approximately 20 minutes and was limited to Document Understanding in the US region. ## Root cause A scheduled backend deployment supporting Document Understanding triggered a rolling update of service instances in the US region. During this process, replacement instances could not start promptly because the required compute resources were temporarily occupied by other workloads. Because the affected service was already handling heavy traffic, the temporary reduction in available instances caused some requests to be rejected with service unavailable errors. ## Detection An automated failed-requests alert detected the issue at 3:19 pm UTC on September 3, 2026. The alert was escalated immediately, and the incident was declared customer-impacting at 3:20 pm UTC. Detection occurred within approximately one minute of the start of customer impact. ## Response By 3:22 pm UTC, responders joined the incident response call and began coordinating mitigation. At 3:34 pm UTC, a public status update confirmed that investigation was in progress. By 3:39 pm UTC, the cause had been identified and mitigation steps were underway. To restore service, the team paused automatic scaling for the affected service, restored it to its full instance count, and temporarily reduced lower-priority workloads to free compute resources. The service returned to full capacity by 3:40 pm UTC, and failed requests dropped to zero. The incident was confirmed mitigated at 3:47 pm UTC and resolved at 4:17 pm UTC after continued monitoring showed the service operating normally. ## Follow up 1. We are pausing further deployments that could trigger the same capacity reduction until a scheduling fix is in place. 2. We are implementing stricter workload placement rules to ensure Document Understanding service instances are not blocked by other workloads during future deployments. 3. Introduce per-tenant GPU usage limits so that a single workload of very large documents cannot saturate shared capacity for other customers. 4. Return accurate retry guidance on overload responses and propagate it through Document Understanding's internal layers, and make the reduced internal retry policy permanent in configuration. 5. Add leading-indicator alerting on request duplication and worker saturation, and an operational runbook for this overload pattern, so it is recognised before customers are affected.