Multiple Regions - IXP - Incorrect Validation Error Message

Update timeline

1. investigating Apr 16, 2026, 05:26 PM UTC

   We're currently seeing an issue where one IXP endpoint is displaying an incorrect validation error message for a small subset of older IXP projects following a recent change. This message is not accurate and can be safely disregarded, though it may temporarily impact parts of the IXP UI. Importantly, IXP ingestion is not affected, and no action is required from customers at this time. Our team is actively working on a hotfix, which will be deployed as soon as possible. We'll share an update once the fix is in place.

2. identified Apr 16, 2026, 06:17 PM UTC

   A hotfix is currently being rolled out to address the incorrect validation error message affecting a small subset of older IXP projects. During this time, some users may continue to briefly see the incorrect message in the IXP UI. As a reminder, this does not impact IXP ingestion, and no customer action is required. We will confirm once the rollout is complete.

3. monitoring Apr 16, 2026, 06:51 PM UTC

   A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. We are actively monitoring to ensure stability.

4. resolved Apr 16, 2026, 07:16 PM UTC

   A hotfix has now been successfully deployed to address the incorrect validation error message affecting a small subset of older IXP projects. The issue is resolved, and we are continuing to monitor.

5. postmortem Apr 21, 2026, 07:33 PM UTC

   ## Customer Impact Between April 16, 2026 at 4:30 pm UTC and April 16, 2026 at 7:22 pm UTC, a subset of customers encountered incorrect validation error messages when accessing older projects in the IXP Communications Mining web application. Specifically, users saw erroneous error messages that did not reflect actual data issues. These messages may have caused confusion or disrupted normal workflows. Although the underlying change had been deployed across all regions, the observable impact was confined to a very small number of legacy datasets in IXP Communications Mining. Data processing and ingestion continued to function normally throughout the incident, and no customer action is required. ## Root Cause A recently deployed change introduced an error in the validation logic for one API endpoint. This change caused the system to display incorrect validation error messages, which surfaced as server errors. The root cause was a misconfiguration in the endpoint's validation rules that did not properly account for legacy project formats. Requests involving these older projects triggered false error messages in the interface. The underlying data processing and ingestion functionality remained unaffected; the issue was isolated to the error-display layer. Recovery was achieved by quickly developing and deploying a targeted hotfix to restore correct validation behavior for affected projects. ## Detection Anomalous error logging was identified at around 4:30 UTC, and reported by a customer around the same time, at which point engineers began investigating. By 5:21 pm UTC, the team confirmed that the errors were limited to legacy datasets and were manifesting as incorrect validation messages in the interface. ## Response Around 5:21 pm UTC, the team identified the issue as a validation logic misconfiguration affecting older projects, confirmed that data ingestion was unaffected, and updated the status page. The hotfix was built and rolled out globally. Full resolution was confirmed at 7:18 pm UTC, when the status page was updated to Resolved. ## Follow-up **Short-term** * Enhance testing to automatically test for behaviour of legacy IXP projects * Enforce consistent serialisation logic across codebase in standard pattern to avoid unexpected edge cases, with focus on testing we can serialise legacy data models that predate later validation requirements