Single zone power event – West Europe

Update timeline

1. postmortem Jan 18, 2023, 01:10 PM UTC

   What happened? Between 09:44 and 13:10 UTC on 18 January 2023, a subset of customers using Storage services in West Europe may have experienced higher than expected latency, timeouts or HTTP 500 errors when accessing data stored on Storage accounts hosted in this region. Other Azure services with dependencies on this specific storage infrastructure may also have experienced impact – including Azure Application Insights, Azure Automation, Azure Container Registry, Azure Database for MySQL, Azure Database for PostgreSQL, Azure Red Hat OpenShift, Azure Search, Azure SQL Database, and Azure Virtual Machines (VMs). What went wrong and why? We determined that an issue occurred during a planned power maintenance. While all server racks have redundant dual feeds, one feed was powered down for maintenance, and a failure in the redundant feed caused a shutdown of the affected racks. This unexpected event was caused by a failure in the electrical systems feeding the affected racks. Two Power Distribution Unit (PDU) breakers tripped, which were feeding the impacted racks. The breakers had lower than designed trip values of 160A, versus the 380A to which they should have been set. Our investigation determined that this lower value was a remnant from previous heat load tests, which should have been aligned back to the site design after testing had completed. This led to an overload event on the breakers once power had been removed from the secondary feeds for maintenance. This caused an incident for a subset of storage and networking infrastructure in one datacenter of one Availability Zone in West Europe. This impacted storage tenants, and network devices which may have rebooted. How did we respond? The issue was detected by the datacenter operation team performing the maintenance at the time. We immediately initiated the maintenance rollback procedure, and restored power to the affected racks. Concurrently, we escalated the incident and engaged other Azure service stakeholders to initiate/validate service recovery. Most impacted resources automatically recovered following the power event, through automated recovery processes. The storage team identified two storage scale units that did not come back online automatically – nodes were not booting properly, as network connectivity was still unavailable. Networking teams were engaged to investigate, and identified a Border Gateway Protocol (BGP) issue. BGP is the standard routing protocol used to exchange routing and reachability information between networks. Since BGP functionality did not recover automatically, 3 of the 20 impacted top-of-rack (ToR) networking switches stayed unavailable. Networking engineers restored the BGP session manually. One storage scale unit was fully recovered by 10:00 UTC, the other storage scale unit was fully recovered by 13:10 UTC. How are we making incidents like this less likely or less impactful? • We have reviewed (and corrected where necessary) all PDU breakers within the facility to align to site design (Completed). • We are ensuring that the Operating Procedures at all datacenter sites are updated to pre-check breaker trip values prior to all maintenance in the future (Estimated completion: February 2023). • We are conducting a full review of our commissioning procedures around heat load testing, to ensure that systems are aligned to site design, after any heat load tests (Estimated completion: February 2023). • In the longer term, we are exploring ways to improve our networking hardware automation, to differentiate between hardware failure and power failure scenarios, to ensure a more seamless recovery during this class of incident (Estimated completion: September 2023). How can customers make incidents like this less impactful? • Consider using Availability Zones (AZs) to run your services across physically separate locations within an Azure region. To help services be more resilient to datacenter-level failures like this one, each AZ provides independent power, networking, and cooling. Many Azure services support zonal, zone-redundant, and/or always-available configurations: https://docs.microsoft.com/azure/availability-zones/az-overview • Consider which are the right Storage redundancy options for your critical applications. Zone redundant storage (ZRS) remains available throughout a zone localized failure, like in this incident. Geo-redundant storage (GRS) enables account level failover in case the primary region endpoint becomes unavailable: https://docs.microsoft.com/azure/storage/common/storage-redundancy • Consider using Azure Chaos Studio to recreate the symptoms of this incident as part of a chaos experiment, to validate the resilience of your Azure applications. Our library of faults includes VM shutdown, network block, and AKS faults that can help to recreate some of the connection difficulties experienced during this incident – for example, by targeting all resources within a single Availability Zone: https://docs.microsoft.com/azure/chaos-studio • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://www.aka.ms/AzPIR/6S_Q-JT8