Datacenter power event - West US 2

Update timeline

1. postmortem Aug 28, 2022, 02:00 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/MMXN-RZ0 What happened? Between 02:47 UTC on 27 Aug 2022 and 02:00 UTC on 28 Aug 2022, a subset of customers experienced failures when trying to access resources hosted in the West US 2 region. Although initially triggered by a utility power outage that affected all of our datacenters in the region, the vast majority of our backup power systems performed as designed to prevent impact. Failures of a small number of backup power systems led to customer impact in two datacenters. Most customers were recovered by 07:00 UTC on 27 Aug 2022, but small subsets of resources required manual recovery – with the final set being brought back online by 02:00 UTC on 28 Aug 2022. During this impact window, several downstream Azure services that were dependent on impacted infrastructure also experienced issues – including Storage, Virtual Machines, App Services, Application Insights, Azure Database for PostgreSQL, Azure Red Hat OpenShift, Azure Search, Azure SQL DB, Backup, Data Explorer, ExpressRoute, and NetApp Files. What went wrong, and why? On August 27 at 02:47 UTC, we identified a power event that caused impact to a number of storage and compute scale units in the West US 2 region. The West US 2 region is made up of 10+ datacenters, spread across three Availability Zones on multiple campuses. During this event, the whole region experienced a utility power outage, impacting all datacenters in the region. A failure on major distribution lines caused at least two substations to lose power. That resulted in loss of utility power across a broad area that included all three Availability Zones in the West US 2 region. In all datacenters except two, our backup power systems performed as designed, transitioning all infrastructure to run briefly on batteries and then on generator power. But in two separate datacenters, two unique but unrelated issues occurred that prevented some of the servers in each datacenter from transitioning to generator power. Since these two datacenters were in two different Availability Zones, customers may have been impacted by both. In the first datacenter, impact was caused when a small number of server rack level Uninterruptible Power Supply (RUPS) systems failed to stay online during the transition to generator, creating a momentary loss of power to the servers. These servers were immediately re-energized once backup generators started and supported the load. In the second datacenter, several Primary UPS systems (approximately 12% of the total UPS systems in the datacenter) failed to support the load during the transition to generator, due to UPS battery failures. As a result, the downstream servers lost power until the UPS faults could be cleared and put back online with utility supply. The initial trigger to this event was when a high voltage static wire (used to help protect transmission lines against lightning strikes) failed. When the static wire failed, it created a voltage surge on the 230kV lines, causing breakers at two substations (approximately 30 miles apart) within the utility power grid to open. The root cause of the static wire failures is still under investigation by the utility provider. How did we respond? This event was first detected by our EPMS (Electrical Power Monitoring System) in West US 2, which in turn notified our datacenter team of the utility loss issue, and then of equipment failure issues. While the vast majority of datacenters transitioned to backup power without issue, two specific datacenters experienced different UPS issues described above that prevented a full transition to backup power sources. Due to the nature of this event, the team followed our Emergency Operations Procedure (EOP) to manually restore Mechanical, Electrical, Plumbing (MEP) equipment to its operational state. Once the MEP was returned to an operational state, the racks began to recover. The Public Utility Department (PUD) was able to close their breakers and restore utility power to our datacenters by 03:48 UTC. This enabled the datacenter teams to begin the recovery of the affected equipment and restoration of power to the impacted racks. By 04:46 UTC, power was fully restored to all affected racks, and services continued their recovery. Four Azure Storage scale units were impacted by the power loss (one Standard, two Premium, one Ultra Disk scale unit) resulting in the data hosted on these becoming inaccessible until power was restored and the scale units recovered to healthy states. The Standard Storage scale unit was fully available by 07:45 UTC, although the vast majority of clients would have seen availability restored by 06:05 UTC. The two Premium Storage scale units were restored by 0510 UTC. Due to a software bug (the fix for which is already in our deployment pipeline) a small subset of disk requests ( 99.9%) was available by 05:15 UTC on 08/27. Impacted Azure compute scale sets were brought back online – mostly automatically after storage recovered, but a subset of infrastructure and customer VMs required manual mitigations to ensure they came back online successfully. VMs that were using the Trusted Launch feature, in particular, did not automatically recover and required engineering team intervention to restore – all of these VMs were restored to a functional state by 00:20 UTC on 8/28. How are we making incidents like this less likely or less impactful? Already completed: • We have completed detailed inspections on our generator systems and all generators are in good operating condition. • UPS systems have been inspected and all components are operating and functioning per design specifications. This inspection highlighted that battery replacement is required. • After reviewing the entire line up, batteries have been replaced in the UPS units that experienced failures. Short term: • Complete deployment of the Storage software fix for the bug that caused a small tail of errors following Premium Storage scale unit recovery. • Complete deployment of the software fix for the trusted VM feature, which caused some VMs not to come back online automatically after storage recovery. Longer term: • We are working on several platform, telemetry, and process improvements that will reduce Ultra Disk replica recovery time. • We are improving our VM migration times to healthy hosts for faster recovery. This includes telemetry improvements to identify long running/stuck migration operations to identify issues more quickly. • Microsoft uses multiple equipment vendors and designs – findings will be reviewed against our global fleet and, where necessary, applied beyond the impacted datacenters. • Process failure mode effects analysis (PFMEA) review of the processes utilized during the event, applying lessons learned and improving our methodology. This includes assessing human touch points, working to engineer out or automate systems for smoother transition or recovery. Findings will also be applied to our Tabletop/GameDay exercises, ensuring team members are familiar and prepared to respond. How can customers and partners make incidents like this less impactful? • While Availability Zones are designed in ways to reduce correlated failures, they can still occur. We encourage customer and partner Business Continuity & Disaster Recovery (BCDR) plans to include the ability to failover between regions, in case of a region-wide incident. While the likelihood of failure decreases with the magnitude of the failure, it never goes to zero: https://docs.microsoft.com/azure/availability-zones/cross-region-replication-azure • Consider which are the right Storage redundancy options for your critical applications. Geo-redundant storage (GRS) enables account level failover in case the primary region endpoint becomes unavailable: https://docs.microsoft.com/azure/storage/common/storage-redundancy • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/en-us/azure/architecture/framework/resiliency • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? We are piloting this “PIR” template as a potential replacement for our “RCA” (Root Cause Analysis) template. You can rate this PIR and provide any feedback using our quick 3-question survey: https://www.aka.ms/AzPIR/MMXN-RZ0