Wide Area Network - Multiple Regions

Update timeline

1. postmortem Jun 29, 2022, 08:14 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/YKDK-TT8 What happened? Across 28 and 29 June, 2022, two independent Azure networking service incidents occurred simultaneously. While very few customer subscriptions were impacted by both, the overlapping nature of the incident timelines meant that we previously delivered a Preliminary Post Incident Review (PIR) that addressed both issues. After thorough investigations, we are publishing a Final PIR for each incident independently. Below is the Final PIR for Incident YKDK-TT8, summarizing our EMEA-impacting Wide Area Network (WAN) issue – we have separately published a Final PIR for Incident YVTL-RS0, summarizing our multi-region Software Load Balancer (SLB) issue. Between 02:40 UTC and 20:14 UTC on 29 June 2022, a subset of customers experienced intermittent network failures and packet loss due to an issue with a router in the Microsoft Wide Area Network (WAN) in the London, UK area. We have determined that one of several WAN routers handling traffic transiting this area experienced a partial hardware failure, which caused it to route improperly 0.4% of the packets it normally handles. This caused intermittent periods of packet loss for a subset of customers whose traffic went through that router, which included a subset of the traffic being served to and from four regions – UK West, UK South, Europe West and Europe North. Some customers using ExpressRoute and/or VPN Gateway were impacted by this, as their traffic traversed this router. Customers affected by this would have experienced impact in the form of lower transmission throughput, long delays in connections and, in the most extreme cases, connection failures. What went wrong, and why? Upon investigating this issue, we identified an issue with part of our WAN infrastructure located in London, U.K.. One specific router in London intermittently started dropping network packets on one line card. This impacted 0.4% of the packets traversing it and caused intermittent network failures for a subset of customer traffic flowing to/from the UK West, UK South, Europe West, and Europe North regions. Since end users experienced this issue intermittently, this was characterized as a ‘gray’ failure so alerting was not immediately able to identify that there was an issue. On this single faulty router, only four of its 140 physical ports had a physical failure, which was causing a section of the router to malfunction intermittently without producing an actionable error message. For Azure VPN Gateway customers, on-premises customer traffic goes through the WAN routers on their path before reaching Azure VPN Gateway, so intermittent packet drops on the core router caused connectivity to be disrupted for Azure VPN scenarios. How did we respond? When first investigating the impact of this issue, we initially invested significant effort attempting to identify software-based causes, because failures were being observed across 17 regions. Router issues typically only impact a small number of regions, and we identified a number of concurrent software changes that had been deployed recently. As the investigation continued, engineers were able to separate the impact from an unrelated Software Load Balancer (SLB) incident, and at approximately 19:15 UTC the remaining impacted traffic was triangulated to WAN routers located near London – with related failures only impacting a small subset of traffic across four regions, not 17. By 20:00 UTC, engineers were able to identify the partially malfunctioning router. Within the next 15 minutes, the engineers removed the unhealthy router from the network – at which point redundant routers took over the traffic, and fully resolved the issue. As mentioned above, this specific partial hardware failure was not producing any specific message in the router that could be tracked or monitored, and the overall low intensity and intermittence of this issue (<0.4% of packets dropped, because of 4 faulty ports out of 140) caused our traffic-quality alerting to miss this issue, so the investigation had to resort to manual inspection of router’s behavior and second-degree signals. How are we making outages like this less likely or less impactful? Already completed: • We enhanced our WAN traffic-quality alerting systems to identify more accurately these low-intensity but long-duration issues, to improve triangulation in future. • We have worked with our hardware vendor to improve their error messages when partial failures like this happen, and are consuming this new signal into our existing automatic alerts and automatic mitigations. Work in progress: • We are replacing the faulty components on the failing London router. This will be brought back into rotation slowly and safely, as we have sufficient redundant capacity with other routers in the network. • We are investing in improved VPN Gateway end-to-end diagnostics to help identify and mitigate this class of issue more quickly. In the longer term: • We are planning enhancements to ExpressRoute traffic-quality monitoring, to better identify impact to individual customers during these kinds of scenarios. How can we make our incident communications more useful? We are piloting this "PIR" template as a potential replacement for our "RCA" (Root Cause Analysis) template. You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/YKDK-TT8