Delayed US - IXP - Model unvailable

Update timeline

1. monitoring Apr 22, 2026, 02:18 PM UTC

   We identified an issue that caused some IXP models to be unavailable or incorrectly shown as “not found.” A fix has been deployed, and we are continuing to monitor the situation closely. We will share a detailed root cause analysis for this incident at a later time.

2. resolved Apr 22, 2026, 03:22 PM UTC

   In the GXP US region, between 13:28:33 UTC and 13:52:42 UTC on April 22, 2026, customers experienced HTTP 404 errors when retrieving extractions for IXP UCD models, resulting in a full outage of runtime extractions during that period. The issue also impacted CM predictions, with additional 404 errors observed . This issue has since been resolved.

3. postmortem Apr 30, 2026, 03:11 PM UTC

   ### Customer Impact Between April 22, 2026 1:28 pm UTC and April 22, 2026 1:52 pm UTC, some customers in the US region were unable to perform document extractions using IXP. During this window, all extraction and prediction requests against pinned model versions returned HTTP 404 Not Found errors. We sincerely apologize for the disruption this caused to your operations. ### Root Cause The incident was triggered by a deployment of a new release of IXP in the impacted region. A targeted code change that required a manual preparation step to be completed before deployment was released before the manual step was run. In environments with more frequent release schedules, this step was handled automatically by an existing part of the deployment process. However, the US and EU regions follow a less frequent, consolidated release schedule, so this automation was not present and the required manual step was missed. As a result, several components of the model management system began reading data from an incorrect location. Because the required data did not exist, requests for extractions and predictions against pinned model versions returned 404 Not Found errors. ### **Detection** Automated monitoring detected the issue after the deployment completed, at approximately 1:28 pm UTC on April 22, 2026. These monitoring checks, which continuously validate core user workflows including extraction against pinned model versions, began failing as soon as the service started returning errors. ### **Response** Upon detection, our engineering team identified the root cause and manually ran the required step against the affected environment, restoring the model management system to read from the correct location. After a sustained monitoring period to verify stable recovery—including confirmation that automated monitoring checks resumed passing and that traffic returned to normal response patterns—the incident was confirmed as fully resolved at 3:22 pm UTC. ### Follow-Up The IXP delayed release process will be reviewed, with a specific focus on why this issue was not seen in earlier regular releases.