Azure Software Load Balancer - Multiple Regions

Update timeline

1. postmortem Jul 01, 2022, 04:00 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/YVTL-RS0 What happened? Across 28 and 29 June, 2022, two independent Azure networking service incidents occurred simultaneously. While very few customer subscriptions were impacted by both, the overlapping nature of the incident timelines meant that we previously delivered a Preliminary Post Incident Review (PIR) that addressed both issues. After thorough investigations, we are publishing a Final PIR for each incident independently. Below is the Final PIR for Incident YVTL-RS0, summarizing our multi-region Software Load Balancer (SLB) issue – we have separately published a Final PIR for Incident YKDK-TT8, summarizing our EMEA-impacting Wide Area Network (WAN) issue. Between 05:26 UTC on 28 June 2022 and 04:00 UTC on 1 July 2022, a subset of customers experienced intermittent network failures and packet loss while trying to connect to resources behind a Software Load Balancer (SLB). This incident impacted customers whose Load Balancer frontend IPs (VIPs) were in either of two specific scenarios: (1) Azure Firewall customers with more than 15 Destination IP addresses (DIPs) behind an Azure Load Balancer, combined with a requirement for both upstream and downstream packets of a single flow to always land on the same DIP. This scenario impacted services in 14 regions. (2) A service endpoint or private link endpoint connection to a service in the region with more than 15 DIPs – for example, SQL and/or Storage. Impacted customers experienced intermittent connectivity failures and packet loss while trying to connect to their resources. This scenario impacted services in three regions. What went wrong, and why? The Software Load Balancer (SLB) is our software-based routing and load balancing service for all network traffic within our datacenters and clusters. At the highest level, it is a set of servers called multiplexers or MUXs that create an SLB scale unit. Each scale unit hosts many VIPs (Virtual IPs) to serve traffic to multiple DIPs (Destination IPs, these are the IPs of the resources behind the VIP). The MUXs then direct traffic to the DIPs that are in service behind the VIP, which are dynamically monitored utilizing periodic probes to monitor health of DIPs. Upon investigating this issue, we identified a change in our latest code deployment which caused some MUX instances to store the backend DIPs in a different order. The change only occurred for load balancers with more than 15 DIPs in the backend pool. The order of the DIPs is important, as it is tied directly to the process used to select the backend server that receives a particular connection. The ordering change impacts traffic that passes through more than one MUX. There are two scenarios where this can occur – the first is when traffic passes through a firewall in both the incoming and return paths; the second is when a particular connection moves from one MUX to another MUX, due a change in routing path through the physical network. The first scenario impacted Azure Firewall customers, but only those with more than 15 firewall instances behind a Load Balancer. That’s because Azure Firewall is a stateful firewall that leverages Azure Load Balancer behind the scenes, scaling horizontally to handle traffic load. Since flows are pinned to instances for firewall processing, the load balancer issue meant that packets for a given flow would be received at different instances and be dropped, impacting customer traffic. The number of instances behind the Firewall’s internal Load Balancer are determined by a customer’s traffic pattern. Not all Firewall’s internal Load Balancers have this many instances, which led to delays in correlation between customer reported issues and the behavior of the platform. The second scenario occurred due to a change in the flow identifier for an IPv6 packet routing header used for Private Link and service endpoint traffic. Traffic that used this technology would have seen drops as the connection moved from one MUX to another MUX. The new MUX would forward the connection to a different server in the backend pool, which would then reject the request. When we deploy updates to our cloud, we use a system called Safe Deployment Practices (SDP). SDP is how we manage change automation so that all code and configuration updates go through well-defined stages, to catch regressions and bugs before they reach customers or – if they do make it past the early stages – impact the smallest number possible. Although telemetry observed a small increase in errors (and a number of customer support requests had been raised) the intermittent nature of this failure meant that our health checks did not correlate these signals to the recent change. This meant that the code had propagated to 14 regions (of our 60+ regions) before we correlated customer impact to this specific deployment. How did we respond? We engaged multiple investigation and mitigation workstreams in parallel. While we worked to understand the issue, we initially used targeted operations to mitigate VIPs manually for specific customers that were identified as impacted. Simultaneously, we developed a hotfix that would address the bug that could be applied without causing further impact to customer traffic. After the hotfix was tested, it was rolled out using a phased approach as per our Safe Deployment Practices. Additionally, we created automation that would scan all VIPs every five minutes and mitigate any VIP that was in an inconsistent state to minimize the impact to customers while the hotfix rolled out. This script continued to run until we had our sustainable mitigation completely rolled out. How are we making outages like this less likely or less impactful? Already completed: • We have restored consistent ordering for our backend instances, returning to the pre-incident state. The order of these is tied directly to the process used to select the backend server that receives a particular connection. Work in progress: • To address SLB scenario #1 above, we are investing in additional monitoring for Azure Firewall to detect cases when upstream and downstream packets on a single flow do not reach the same DIP. • To address SLB scenario #2 above, we are fixing the IPv6 flow identifier change so that it remains consistent for the lifetime of the connection and, as a result, all packets will land on a single MUX. • For Azure Firewall, we are investing in improved packet capture support specific to firewalls (which will help to identify and mitigate issues like this more quickly in the future), as well as enhanced logging for TCP flows (to help isolate the issue more effectively, by logging invalid packets, RST, FIN and FIN-ACK packets). In the longer term: • This incident has validated the need for more thorough end-to-end monitoring during safe deployments, so we are accelerating the onboarding of additional Azure services to use our internal Gandalf AIOps solution suite (See: https://azure.microsoft.com/blog/advancing-safe-deployment-with-aiops-introducing-gandalf/) How can we make our incident communications more useful? We are piloting this "PIR" template as a potential replacement for our "RCA" (Root Cause Analysis) template. You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/YVTL-RS0