Multiple Regions - IXP - [Some customers are unable to access IXP]

Update timeline

1. investigating Mar 03, 2026, 08:29 PM UTC

   Affected Regions: - Europe - India - UK - Switzerland

2. identified Mar 03, 2026, 09:36 PM UTC

   Identified issues and team is working towards resolution.

3. identified Mar 03, 2026, 09:37 PM UTC

   Identified issues and team is working towards resolution.

4. monitoring Mar 03, 2026, 10:31 PM UTC

   A fix is deployed and we are monitoring the fix.

5. resolved Mar 03, 2026, 11:36 PM UTC

   A fix is deployed and we are monitoring the fix.

6. postmortem Mar 05, 2026, 11:59 AM UTC

   ## Customer impact Between March 3, 2026, at 3:20 pm UTC and March 3, 2026, at 11:49 pm UTC, some customers were unable to access the IXP Platform. During this period, users who were making cross-region requests to IXP were receiving "Not Found" responses. This encompassed the design-time UI, the API, and consumers via Activities. The disruption lasted approximately eight hours and 27 minutes. ## Root cause The incident was triggered by a configuration change to the service routing layer for IXP, specifically a modification to the service routing configuration deployed on March 3, 2026, at 3:20 pm UTC. This change was intended to prepare for an upcoming feature release but inadvertently disrupted cross-region request handling. As a result, requests originating from one region and targeting IXP in another region were blocked or misrouted, causing service unavailability for affected customers. Attempts to roll back the change initially worsened the issue, as the rollback did not restore the previous stable state and introduced additional routing inconsistencies. Recovery required building and deploying a targeted hotfix to restore correct routing behavior across all impacted regions. ## Detection The incident was first detected on March 3, 2026, at 6:21 pm UTC, when clients advised that they were unable to access the IXP design-time UI. Automated monitoring confirmed an issue with cross-region requests since the deployment. The issue was classified as an incident and escalated within minutes, with the status page updated to reflect the outage. Initial investigation began immediately, with engineering teams coordinating in real time. ## Response On March 3, 2026, at 6:21 pm UTC, engineering teams began investigating the root cause, focusing on recent configuration changes to the service routing layer. An initial attempt to fix the issue by reverting a minimal set of changes was unsuccessful at 7:55 pm UTC, and the engineering team started to work on a more robust fix. By 9:07 pm UTC, the issue was identified as a problematic service routing configuration deployed earlier that afternoon. A hotfix was developed and tested in a smaller environment, then deployed to the Japan region to verify resolution. By 10:00 pm UTC, the hotfix was pushed to the EU and US regions, where the majority of affected customers were located. Full recovery was achieved by March 3, 2026, at 11:49 pm UTC, when monitoring confirmed normal operation and customers reported restored access to IXP. The status page was updated to resolved, and no further customer cases were reported. Recovery was verified through automated checks, customer confirmations, and monitoring data, ensuring that cross-region requests to IXP were functioning as expected. ## Follow up To prevent similar incidents, we are implementing several technical and process improvements: * Automated validation checks will be added to pre-production environments, ensuring that changes to service routing layers are tested for cross-region compatibility before rollout. * Enhanced monitoring will be deployed to detect anomalies in cross-region request handling, providing earlier warning and more granular visibility into service health. * We will improve how quickly and precisely we can revert recently deployed changes so that issues can be resolved as soon as possible once they have been diagnosed. These measures build on lessons learned from similar past events, such as the incident involving email delivery disruptions due to configuration changes. We are committed to systematically addressing this pattern by strengthening our deployment safeguards, improving detection capabilities, and ensuring rapid, reliable recovery for our customers. Technical improvements will be rolled out over the coming weeks, with ongoing review to ensure effectiveness and resilience.