Microsoft Azure experienced a major incident on October 26, 2022, lasting 5h 35m. The incident has been resolved; the full update timeline is below.
Update timeline
- postmortem Oct 26, 2022, 06:00 AM UTC
What happened? Between 00:25 UTC and 06:00 UTC on 26 October 2022, a subset of customers using Azure Cosmos DB in the East US region may have experienced issues connecting to the service. Connections to Azure Cosmos DB accounts in this region may have resulted in an error or timeout. Downstream Azure services that rely on Azure Cosmos DB also experienced impact during this window - including Azure Application Insights, Azure Automation, Azure Container Registry, Azure Digital Twins, Azure Policy, Azure Rights Management, Azure Red Hat OpenShift, and Azure Spatial Anchors. What went wrong, and why? A change to the front-end gateway of Azure Cosmos DB, to include additional diagnostic information, was introduced on the affected cluster earlier this month. The change had no effect to the fidelity of the system since introduction. On 26 October, a configuration change was applied to the Azure Load Balancer. This change resulted in intermittent network connectivity issues, from which the system can normally recover. However, the diagnostic change resulted in higher-than-expected time spent in the Kernel, which resulted in spikes of high CPU utilization across the cluster. In turn, this created an increase in exceptions, and even larger time spent locking in the kernel. This resulted in timeouts and increased latency of incoming requests, ultimately leading to the customer impact described above. How did we respond? Our monitors alerted us to the impact on this cluster. We worked with our customers and partners to trigger mitigation steps while investigating the factors contributing to this issue. To mitigate the incident, accounts were offloaded from the impacted cluster to other clusters in the same region. To safely offload the quantity of accounts we had to migrate, we systematically moved each database account to an alternative healthy cluster. As a result, the lower load improved the state of the impacted cluster and enabled recovery. All customer impact was confirmed mitigated by 06:00 UTC. How are we making incidents like this less likely or less impactful? • We have paused the configuration changes for the Azure Load Balancer. (Completed) • We have worked to fix the original regression in the diagnostic stack. (Completed) • We are improving our load balancing automation, to speed up recovery in similar circumstances. (Estimated completion: November 2022). How can customers make incidents like this less impactful? • Consider configuring your accounts to be globally distributed – enabling multi-region for critical accounts would allow for a customer-initiated failover during regional service incidents like this one. For more details, refer to: https://docs.microsoft.com/azure/cosmos-db/distribute-data-globally • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/en-us/azure/architecture/framework/resiliency • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://www.aka.ms/AzPIR/JKWW-JP8