US - Document Understanding - Degraded Performance

Update timeline

1. investigating Jul 30, 2026, 10:44 PM UTC

   We are investigating reports of degraded performance impacting Extraction and Classifications for Document Understanding in US. Our teams are working to identify the cause and will share more details as the investigation progresses.

2. monitoring Jul 30, 2026, 11:48 PM UTC

   We have implemented a fix for the degraded performance impacting Extraction and Classification for Document Understanding in US. We are monitoring the results and will provide an update shortly.

3. monitoring Jul 31, 2026, 01:08 AM UTC

   We are continuing to monitor the fix. Performance and error rates have returned to normal levels. We will provide an update once we confirm full recovery.

4. resolved Jul 31, 2026, 01:28 AM UTC

   This incident has been resolved. Extraction and Classification for Document Understanding in US are operating normally.

5. postmortem Aug 06, 2026, 02:04 PM UTC

   ## Customer impact Between July 30, 2026 at 10:00 pm UTC and July 31, 2026 at 1:28 am UTC, a subset of customers using Helix models for Document Understanding in the US experienced degraded performance, including failed requests returning HTTP 503 errors and client-side timeouts. The impact lasted approximately 3 hours and 28 minutes. Some requests continued to process successfully, but affected customers may have seen retries, delayed processing, or failed document extraction and classification jobs. ## Root cause The root cause was duplicate re-processing of the same document pages during a period of high retries. Successful processing results were cached; when the queue was full, failed submissions were not cached. Repeated retries re-entered processing instead of being deduplicated. During the incident window, duplicate processing accounted for the majority of compute work, which drove sustained resource saturation and caused additional 503 responses - which increased load and led to additional 503 responses. ## Detection Automated alerting detected the first queue overflow errors began around 10:00 pm UTC. ## Response At 10:44 pm UTC, we posted an investigation notice for degraded Extraction and Classification performance in the US. By 11:26 pm UTC, we freed capacity by scaling down an unused processing path and freeing additional capacity for the affected Document Understanding workload. ## Follow up 1. We plan to implement short-lived failure caching for repeated document submissions, so that duplicate retries after a queue overflow are rejected early instead of re-entering expensive processing. 2. We are improving monitoring and alerting to detect this class of issue sooner.