IXP Communications Mining elevated error rate in US region

Update timeline

1. resolved Aug 18, 2026, 10:58 AM UTC

   An abnormal usage pattern has caused increased load in IXP Communications Mining's API, impacting API responses across the product with elevated 500 error rates. Limited to a number of customers in the US.

2. postmortem Aug 18, 2026, 09:25 PM UTC

   ## Customer Impact On August 12, 2026, between approximately 15:10 and 18:15 UTC, Communications Mining (IXP) users in the United States region experienced intermittent request failures. The incident was confined to one of the region's deployment units, where users on it saw failures in bursts of five to ten minutes, with up to 5–7% of their requests failing with 5xx errors at peak. Between bursts the service operated normally, retried requests generally succeeded, and no data was lost. ## Root Cause A storm of requests to a rarely used API feature that computes machine-learning predictions on demand coincided with repeated retraining of the requested model. Each retraining invalidated cached predictions, turning each request into a multi-minute computation. The API placed no time limit on how long a request could wait for this computation, so these long-running requests progressively occupied all request-processing capacity, causing unrelated requests to fail. Automatic scaling reached its maximum capacity quickly and could not compensate. ## Detection Automated monitoring detected the failures at 15:15 UTC, about five minutes after impact began, and paged the on-call engineer. The incident was initially attributed only to the client generating the request storm, but client reports and further investigation showed a wider set of users was affected during the failure bursts. ## Response The on-call engineer traced the failures to the unbounded wait in the on-demand prediction path. Service capacity was repeatedly restored by automatic instance replacement while a code fix was developed. The fix is a strict timeout on the contributing request so it fails fast without affecting other requests, and was deployed to the affected region at approximately 18:00 UTC, and resolution was confirmed at 18:15 UTC. ## Follow-Up 1. The strict timeout and load-shedding fix has been made permanent and released to all regions (completed August 13, 2026). 2. Evaluate per-client limits on on-demand prediction computation so that a single client's usage cannot degrade the shared API.