Europe - Document Understanding - Degraded Performance

Update timeline

1. investigating Jul 29, 2026, 11:54 AM UTC

   We are investigating reports of degraded performance impacting Extraction and Classifications for Document Understanding in Europe. Our teams are working to identify the cause and will share more details as the investigation progresses.

2. investigating Jul 29, 2026, 12:05 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jul 29, 2026, 12:57 PM UTC

   We have applied the fix, we are currently monitoring the situation closely.

4. resolved Jul 29, 2026, 02:04 PM UTC

   The fix has been successfully applied, and our monitoring confirms that Extraction and Classifications for Document Understanding in Europe are operating normally. This incident has been resolved.

5. postmortem Jul 31, 2026, 10:39 AM UTC

   ## Customer impact Between July 22 and July 29, 2026, customers using Document Understanding in the Europe region experienced intermittent delays and, in a limited number of cases, failed document operations. The issue affected document extraction and splitting/classification operations using Helix 2.0-based models— other model types were not affected. The impact occurred in three windows, each detected by our automated monitoring: * July 22, approximately 5:10 pm to 7:39 pm UTC * July 24, approximately 7:45 am to 3:01 pm UTC * July 29, approximately 10:55 am to 2:05 pm UTC During these windows, operations that normally complete in under a minute could take several minutes, and a small number of operations failed after all retry attempts. Between the first two windows, a low rate of intermittent delays persisted. The issue was isolated to the Europe region and did not affect other geographies. ## Root cause The root cause was a concurrency issue in the inference service that runs these models. Two stages of processing—preparing incoming requests and generating results—shared an internal component that only one operation can use at a time. Under high-load, operations queued for this shared component, slowing result generation even though compute capacity was available. Slow operations kept their processing slots occupied, so new requests queued behind them and some were rejected, and a small number of operations ultimately failed after exhausting all retry attempts. Large, image-heavy documents increased use of the shared component and made the slowdown worse. Retried requests for the same document are automatically deduplicated, so retries did not multiply the processing work; however, rejected requests did not tell clients when to retry, which added some pressure during the constrained periods. The July 29 recurrence happened because a protective configuration change applied after the earlier occurrences was unintentionally reverted during a routine deployment. It was re-applied the same day and has now been made permanent. ## Detection Automated alerts detected the initial issue at 5:14 pm UTC on July 22, approximately four minutes after failures began to rise. Monitoring showed elevated failed requests and longer processing times. After the first window was resolved, our engineering team continued to track a low rate of intermittent delays through end-to-end tests and telemetry reviews while the investigation continued. A new automated alert detected the July 24 recurrence, and the July 29 recurrence was detected by automated alerts within minutes of onset. ## Response On July 22, automated capacity scaling restored throughput and the failure rate returned to near-baseline levels. The status page was updated at 5:45 pm UTC, and after continued monitoring showed no further customer-visible failures, the incident was closed at 7:39 pm UTC. After closure, our engineering team continued investigating the underlying cause—that investigation was still in progress when the issue recurred on July 24. On July 24, following the recurrence, our engineering team restarted the affected inference workers and applied a protective configuration change that reduces concurrent processing per worker and increases minimum capacity. The status page was updated at 9:22 am UTC, and the incident was resolved at 3:01 pm UTC after extended monitoring. On July 29, engineers identified that the protective configuration had been unintentionally reverted, re-applied it, and restarted the affected workers. The status page was updated at 11:54 am UTC. Service was confirmed stable—including by affected customers—and the incident was resolved at 2:05 pm UTC. ## Follow-up We are taking the following steps to prevent recurrence and improve resilience: 1. **Retry handling:** Rejected requests now include explicit retry-timing guidance so that clients back off appropriately during constrained periods. Requests for the same document remain automatically deduplicated, so retries do not multiply processing work. 2. **Contention fix:** We have completed a code change that removes the internal contention that caused slow processing under high loads. It is being rolled out and validated under production traffic. 3. **Configuration safeguards:** The protective configuration has been made permanent in our deployment configuration, with change controls in place so that it cannot be unintentionally reverted. 4. **Observability and alerting:** We are improving monitoring and alerting to detect this class of issue sooner, and tightening incident-closure criteria so that incidents are not closed while residual degradation remains.