IXP - Multiple Regions - Partial outage for GPT models

Update timeline

1. investigating May 29, 2026, 06:00 PM UTC

   We are investigating failures and increased latency with GPT models for IXP. Impact: Users may be unable to use GPT models in IXP. Next update: Our teams are working to understand the cause and scope and will share updates as available.

2. monitoring May 29, 2026, 06:27 PM UTC

   We have rolled back a previous change which was responsible for the issue, and are seeing calls to GPT models succeed again. Next update: we expect no further service disruptions, and will update once we have confirmed all regions are recovered

3. resolved May 29, 2026, 07:26 PM UTC

   The incident is resolved

4. postmortem Jun 08, 2026, 02:27 PM UTC

   ## Customer Impact Between May 23, 2026 and May 29, 2026, a subset of customers globally experienced intermittent issues when using Google Gemini models. The affected functionality was part of the IXP-UCD workflow — specifically the document extraction, intelligent preprocessing, and attribution stages — when configured to use Gemini models. Customers may have experienced increased processing times or failures indicating that content could not be retrieved for processing. On May 29, 2026, a smaller number of customers also experienced a related GPT service disruption, which stemmed from the remediation applied for the Gemini issue. Both issues were resolved on May 29, 2026. ## Root Cause The issue was caused by failures in the process used to retrieve document content for Gemini model requests. Under certain conditions, requests to retrieve content were rejected by the model provider, preventing documents from being processed successfully. This resulted in delays and intermittent failures for the IXP-UCD workflow when using affected Gemini models. A change was implemented to use an alternative content delivery method, which eliminated the Gemini failures. While applying this remediation, a configuration issue briefly disrupted GPT model service; this was identified and corrected the same day. ## Detection The issue was identified through a review of service metrics showing elevated processing delays and customer impact. During the investigation, we identified a monitoring gap that prevented expected alerts from notifying engineering teams as service performance degraded. This delayed escalation and response. ## Response After identifying the issue, the team worked with the model provider to investigate the failures and implement a mitigation. A temporary fix was deployed globally, restoring normal service for affected Gemini workloads. Additional validation and deployment activities were completed across all regions to ensure full recovery. The related GPT disruption, which stemmed from the Gemini remediation, was identified and corrected on May 29, 2026, restoring normal GPT service shortly afterward. ## Follow-Up We are implementing the following improvements: **Short-term improvements** * Address the monitoring gap to ensure elevated processing delays trigger alerts as expected.