Networking fiber cut - West Europe

Update timeline

1. postmortem Jul 05, 2023, 04:00 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/FVHB-188 What happened? Between approximately 07:22 UTC and 16:00 UTC on 5 July 2023, Azure customers using the West Europe region may have experienced packet drops, timeouts, and/or increased latency. This impact resulted from a fiber cut caused by severe weather conditions in the Netherlands. The West Europe region has multiple datacenters and is designed with four independent fiber paths for the traffic that flows between datacenters. In this incident, one of the four major paths was cut, which resulted in congestive packet loss when traffic on the remaining links exceeded their capacity. Downstream Azure services dependent on this intra-region network connectivity were also impacted – including Azure App Services, Azure Application Insights, Azure Data Explorer, Azure Database for MySQL, Azure Databricks (which also experienced impact in North Europe, as a result of a control plane dependency), Azure Digital Twins, Azure HDInsight, Azure Kubernetes Service, Azure Log Analytics, Azure Monitor, Azure NetApp Files, Azure Resource Graph, Azure Site Recovery, Azure Service Bus, Azure SQL DB, Azure Storage, and Azure Virtual Machines – as well as subsets of Microsoft 365, Microsoft Power Platform, and Microsoft Sentinel services. What went wrong and why? Due to a fiber cut caused by severe weather conditions in the Netherlands, 25% of network links between two campuses of West Europe datacenters became unavailable. These links were already running at higher utilization than our design target, and there was a capacity augment project in progress to address this. Due to a previous incident related to this capacity augment on 16 June 2023 (Tracking ID VLB8-1Z0), the augment work was proceeding with extreme caution and was still in progress when the fiber cut occurred. As a result of the fiber cut and the higher utilization, congestion increased to a point where intermittent packet drops occurred in many of the intra-region paths. This primarily impacted network traffic between Availability Zones within the West Europe region, not traffic to and from the region itself. As a result of this interruption, Azure services that rely on internal communications with other services within the region may have experienced degraded performance, manifesting in the issues described above. How did we respond? Network alerting services indicated a fiber cut at 07:22 UTC and a congestion alert triggered at 07:46 UTC. Our networking on-call engineers engaged and began to investigate. Two parallel workstreams were spun up to mitigate impact: The first workstream focused on reducing traffic in the region and balancing it across the remaining links. This balancing activity requires a detailed before-and-after traffic simulation to ensure safety, and these simulations were initiated as a first step. At 10:00 UTC we initiated throttling and migration of internal service traffic away from the region. We also started work on rebalancing traffic away from congested links. As a result of these activities, by 14:52 UTC packet drops were reduced significantly, by 15:30 UTC many internal and external services saw signs of recovery, and by 16:00 UTC packet drops had returned to pre-incident levels. We continued to work on reducing high link utilization and by 16:21 UTC the rebalancing activity was completed. The second workstream focused on repairing the impacted links, in partnership with our dark fiber provider in the Netherlands. These cable repairs took longer than expected since access to the impacted area was hindered by the weather and hazardous working conditions. Partial restoration was confirmed by 19:30 UTC, and full restoration was confirmed by 20:50 UTC. While this restored the network capacity between datacenters, we continued to monitor our network infrastructure and capacity before declaring the incident mitigated at 22:45 UTC. How are we making incidents like this less likely or less impactful? • We have repaired the impacted networking links, in partnership with our dark fiber provider in the Netherlands. (Completed) • Within 24 hours of the incident being mitigated we brought additional capacity online, on the impacted network path. (Completed) • Within a week of the incident, we are 90% complete with our capacity augments that will double capacity in our West Europe region to bring utilization within our design targets. (Estimated completion: July 2023) • As committed in a previous Post Incident Review (PIR), we are working towards auto-declaring regional incidents to ensure customers get notified more quickly (Estimated completion: August 2023). How can customers make incidents like this less impactful? • During the incident, we advised customers who were able to fail out of the West Europe region to consider doing so. Customers should consider a multi-region geodiversity strategy for mission-critical workloads, to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/training/modules/design-a-geographically-distributed-application/ and https://learn.microsoft.com/azure/architecture/patterns/geodes • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/FVHB-188