Microsoft Outage History

Started Feb 07, 2026, 07:58 AM UTC· resolved Feb 08, 2026, 04:24 AM UTC

1. postmortem Feb 08, 2026, 04:24 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/_SVS-5_G What happened? Between 07:58 UTC on 07 February 2026 and 04:24 UTC on 08 February 2026, impacted customers experienced intermittent service unavailability, timeouts and/or higher than normal latency for services in the West US region. The event began following a power interruption affecting one of the datacenters within the region, after which impact manifested as infrastructure availability loss and service disruptions across multiple dependent workloads in the region. As power stabilization progressed, recovery proceeded in phases – however, a subset of storage and compute infrastructure components did not return to a healthy state immediately after power restoration, which slowed recovery for dependent components, and contributed to ongoing symptoms such as delayed telemetry and resource recovery. Impacted services included: • Azure AI Search: Customers using Azure AI Search in West US may have experienced query failures, indexing delays, or temporarily unavailable search endpoints. • Azure App Service: Between 07:58 UTC and approximately 09:45 UTC on 07 February 2026, customers using Azure App Service in West US may have experienced HTTP 500 errors, request timeouts, or failed deployments. • Azure Backup: During the impact window, backup jobs in West US may have experienced delays or temporary failures due to storage unavailability. • Azure Cache for Redis: Customers may have observed connectivity failures, increased latency, or transient cache unavailability in West US. • Azure Container Registry: Customers may have experienced delays or failures when pulling or pushing container images in West US. • Azure Cosmos DB: Customers may have experienced a degradation in service availability and/or request latency. Some requests may have resulted in server errors or timeouts. • Azure Data Factory: Customers may have observed pipeline execution failures or delays in West US. Affected integration runtimes recovered following restoration of underlying networking and storage services. • Azure Database for MySQL – Flexible Server: Between 09:40 UTC and 20:50 UTC on 07 February 2026, customers may have experienced connection failures or degraded database availability in West US. • Azure Database for PostgreSQL – Flexible Server: Customers may have experienced database connectivity interruptions or degraded performance in West US. • Azure Databricks: Customers may have experienced workspace unavailability, cluster startup failures, or job execution interruptions in West US. • Azure DevOps: Customers may have experienced delays in log ingestion and analytics reporting in West US. • Azure Event Hubs: Between 07:58 UTC and 19:05 UTC on 7 February 2026, customers may have experienced failures or high latency when sending and processing events. Control plane operations like creating/updating/deleting Event Hubs or Consumer Groups were also impacted. • Azure IoT Hub: Customers may have observed delayed message processing or temporary endpoint unavailability in West US. • Azure Kubernetes Service (AKS): Between 07:58 UTC and 21:46 UTC on 07 February 2026, customers using AKS in West US may have experienced control plane instability, node health reporting delays, or pod scheduling failures. • Azure Key Vault: Customers may have experienced intermittent authentication or secret retrieval failures in West US. • Azure Monitor: Customers may have observed delayed metrics, log ingestion latency, or temporary alerting gaps in West US. • Azure Monitor (Application Insights): Between 07:58 UTC on 07 February 2026 and 04:24 UTC on 08 February 2026, customers may have experienced delayed telemetry ingestion and incomplete monitoring data in West US. • Azure Service Bus: Between 07:58 UTC and 19:05 UTC on 7 February 2026, customers may have experienced transient messaging failures, delayed message processing, or control plane disruptions impacting queues and topics in West US. • Azure SQL Database: Customers may have experienced SQL error 40613 (database unavailable) or connectivity interruptions in West US. Service mitigation included database failover and host recovery. • Azure Storage: Customers may have experienced read/write failures, increased latency, or temporary storage account unavailability in West US. Storage services were restored in phases as power and network infrastructure stabilized. • Azure Stream Analytics: Customers may be experiencing service availability disruptions, including delays or failures in data ingestion, processing, or output delivery. • Azure Virtual Machines: Customers may have experienced VM start failures, allocation errors, or temporary unavailability of running instances in West US. • Azure Virtual Machines (Confidential VM types): Customers may have experienced VM deployment or startup failures in West US, due to dependent storage disruption. • Microsoft Defender for Cloud Apps: Customers may have observed delayed alerts or telemetry ingestion in West US during the recovery period. What went wrong and why? Under normal operating conditions, Azure datacenter infrastructure is designed to tolerate utility power disturbances through redundant electrical systems and automated failover to on-site generators. During this incident, although utility power was still active, an electrical failure in an onsite transformer resulted in the loss of utility power to the datacenter. Although generators started as designed, a cascading failure within a control system prevented the automated transfer of load from utility power to generator power. As a result, Uninterruptible Power Supply (UPS) batteries carried the load for several minutes – until they were fully depleted, leading to customer impact from this power loss as early as 07:58 UTC. Once power loss occurred, recovery was not uniform across all dependent infrastructure. Our datacenter operations team restored power to IT racks by leveraging our onsite generators, with 90% of IT racks powered by 09:31 UTC on 07 February. While infrastructure generally came back online as expected after power restoration, subsets of networking, storage and compute infrastructure required further electrical control system troubleshooting, before power could be restored. We were fully restored and running generator-backed power by 11:29 UTC. Power restoration was sequenced to avoid destabilizing power that had already been brought back online, and due to the inconsistent state of the control system. Following power restoration at the datacenter, recovery of customer workloads progressed in stages based on dependencies. Network and Storage recovery were critical requirements, as compute hosts must be able to access persistent storage to complete boot, validate system state, and reattach customer data before workloads can resume. Network devices and services recovered as expected once power was restored. The power loss affected six storage scale units within the datacenter, of which four recovered as expected once power was restored. Two storage scale units experienced prolonged recovery, which became a primary factor contributing to extended impact as these contained a large subset of storage nodes that failed to complete the boot process. During the boot process, critical artifacts need to be downloaded, but these were delayed or timed out – resulting in these nodes entering a state for which there was not an automated recovery action. Manual operations were attempted, and eventually succeeded in recovering the necessary nodes to bring these storage scale units back online. Due to the dependencies that many compute and platform services have on these storage scale units, there was a delay in overall service restoration. Storage recovery completed by 19:05 UTC. Compute recovery began once power and storage dependencies were partially restored. Automated recovery systems were available and functioning, but the scale of the event resulted in a surge of required recovery activities across the datacenter. This placed sustained pressure on regional shared infrastructure components responsible for coordinating health checks, power state validation, and repair actions. During the initial recovery window, this elevated load reduced the speed at which individual compute nodes could be recovered. While many nodes returned to service as expected, some nodes required repeated recovery attempts, due to insufficient prioritization across concurrent fault categories during the surge of recovery activities. We introduced optimizations to accelerate recovery, including deprioritizing non-critical recovery activities. As pressure on these shared systems decreased, compute recovery progressed more predictably and remaining nodes were gradually restored. Compute recovery completed by 23:30 UTC. How did we respond? • 07:54 UTC on 07 February 2026 – Initial electrical failure in an onsite transformer, although various UPS batteries carried the load for several minutes. • 07:58 UTC on 07 February 2026 – Customers began experiencing unavailability and delayed monitoring/log data. • 08:07 UTC on 07 February 2026 – Datacenter and engineering teams engaged and initiated coordinated investigation across power, network, storage, and compute recovery workstreams. • 09:31 UTC on 07 February 2026 – 90% of facility power restored on generators. • 11:26 UTC on 07 February 2026 – 93% of facility power restored on generators. • 11:29 UTC on 07 February 2026 – 100% of facility power restored on generators with continued recovery of hosted services. • 12:15 UTC on 07 February 2026 – Storage recovery started and dependent services started seeing recovery. • 15:00 UTC on 07 February 2026 – Targeted remediation actions continued for remaining unhealthy services. • 15:25 UTC on 07 February 2026 – Applied configuration changes to optimize processing of recovery events, to assist in expediting the recovery of compute services. • 19:05 UTC on 07 February 2026 – Storage recovery was complete. • 19:24 UTC on 07 February 2026 – Compute recovery system stabilized and fully operational, continued gradually restoring remaining nodes. • 23:30 UTC on 07 February 2026 – Long tail of Compute nodes impact mitigated. • 04:24 UTC on 08 February 2026 – All affected services confirmed mitigated. • 03:42 UTC on 09 February 2026 – Power was transitioned back to utility power, restoring the datacenter to normal operations. How are we making incidents like this less likely or less impactful? • The control system and dependent components have been inspected, reconfigured, and validated to support transfer-to-generator events more reliably. (Completed) • We have reevaluated how best to prioritize critical infrastructure earlier in the power restoration sequence, to ensure a smoother recovery across dependent services. (Completed) • We are working to enhance the recovery system through prioritization and throttling, to improve stability during surge scenarios like this one. (Estimated completion: July 2026) • Finally, we are continually refining how our infrastructure and services come back online after datacenter power loss events, to make these more robust to different failure scenarios. How can customers make incidents like this less impactful? • Since Availability Zones are not available in the West US region, customers should consider a multi-region geodiversity strategy for mission-critical workloads: https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones#deployment-approach-4-multi-region-deployments and https://learn.microsoft.com/azure/well-architected/design-guides/disaster-recovery • Consider leveraging data redundancy options such as Read-Access Geo-Redundant Storage (RA-GRS) to ensure data accessibility during regional incidents: https://learn.microsoft.com/azure/storage/common/storage-redundancy • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/_SVS-5_G

Read the full incident report →

Started Feb 02, 2026, 06:03 PM UTC· resolved Feb 03, 2026, 12:30 AM UTC

1. postmortem Feb 03, 2026, 12:30 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/FNJ8-VQZ What happened? Between 18:03 UTC on 02 February 2026 and approximately 00:30 UTC on 03 February 2026, a platform issue caused some customers to experience degraded performance and control plane failures, for multiple Azure services across multiple regions. Impacted services included: • Azure Virtual Machines (VMs) – Customers may have experienced failures when deploying or scaling virtual machines, including errors during provisioning and lifecycle operations. • Azure Virtual Machine Scale Sets (VMSS) – Customers may have experienced failures when scaling instances or applying configuration changes. • Azure Kubernetes Service (AKS) – Customers may have experienced failures in node provisioning and extension installation. • Azure DevOps (ADO) and GitHub Actions – Customers may have experienced pipeline failures when tasks required VM extensions or related packages. Actions jobs queued and timed out while waiting to acquire a hosted runner. Other GitHub features that leverage this compute infrastructure were similarly impacted, including Copilot Coding Agent, Copilot Code Review, CodeQL, Dependabot, GitHub Enterprise Importer, and Pages. All regions and runner types were impacted. • Other dependent services – Customers may have experienced degraded performance or failures in operations that required downloading extension packages from Microsoft-managed storage accounts, including Azure Arc enabled servers, and Azure Database for PostgreSQL. As part of our recovery, between 00:08 UTC and 06:05 UTC on 03 February 2026, customers using the 'Managed identities for Azure resources' service (formerly Managed Service Identity) may have experienced failures when attempting to create, update, or delete Azure resources, or acquire Managed Identity tokens – in the East US and West US regions. Originally communicated on incident tracking ID _M5B-9RZ, this impacted the creation and management of Azure resources with assigned managed identities – including but not limited to Azure AI Video Indexer, Azure Chaos Studio, Azure Container Apps, Azure Database for PostgreSQL Flexible Servers, Azure Databricks, Azure Kubernetes Service, Azure Stream Analytics, Azure Synapse Analytics, and Microsoft Copilot Studio. Newly created resources of these types may have also failed to authenticate using managed identity. What went wrong, and why? On 02 February 2026, a remediation workflow executed based on a resource policy, intended to disable anonymous access on Microsoft-managed storage accounts. The purpose of this policy is to reduce unintended exposure by ensuring that anonymous access is not enabled unless explicitly required. Due to a data synchronization problem in the targeting logic, the policy was incorrectly applied to a subset of storage accounts that are intentionally configured to allow anonymous read access for platform functionality. These storage accounts form part of the VM extension package storage layer. VM extensions are small applications used during provisioning and lifecycle operations to configure, secure, and manage VMs. Extension artifacts are retrieved directly from storage accounts, during early-stage provisioning and scaling workflows. When anonymous access was disabled across these storage accounts, VMs and dependent services were unable to retrieve required extension artifacts. This resulted in control plane failures and degraded performance across affected services. Although the policy process followed safe deployment practices and was initially rolled out region by region, the health assessment defect caused the incorrect targeting state to propagate broadly across public cloud regions in a short time window. This accelerated the impact beyond the intended blast radius. Once identified, the policy job was immediately disabled to prevent further changes. Rollback procedures to restore anonymous access were validated in a test region and then executed region by region across affected storage accounts. As access was restored, control plane operations gradually recovered and service management queues began to clear. As storage access was restored, a secondary issue emerged. The re-enablement of the VM extension package storage layer allowed multiple previously blocked workflows to resume concurrently, across infrastructure orchestration systems and dependent services. The resulting surge in resource management operations generated a significant and sudden increase in requests to the backend service supporting Managed Identities for Azure resources. As a result, starting at 00:08 UTC on 03 February 2026, customers in East US and West US experienced failures when attempting to create, update, or delete Azure resources that use managed identities, as well as when acquiring Managed Identity tokens. Automatic retry behaviors in upstream components, designed to provide resiliency under transient failure conditions, amplified traffic against the managed identity backend. The cumulative effect exceeded service limits in East US. Resilient routing mechanisms subsequently directed retry traffic to West US, resulting in similar saturation. Although the managed identity infrastructure was scaled out, additional capacity alone did not immediately mitigate impact – because retry amplification continued to generate load at a rate that exceeded recovery capacity. Active load shedding and controlled reintroduction of traffic were required to stabilize the service and allow backlog processing while maintaining normal operations, with full recovery by 06:05 UTC. How did we respond? • 18:03 UTC on 02 February 2026 – Customer impact began, triggered by the periodic remediation workflow starting. • 18:29 UTC on 02 February 2026 – Internal service monitoring detected a subset of regions having an increasing number of control plane failures. • 19:46 UTC on 02 February 2026 – Correlated issues across multiple regions. • 19:55 UTC on 02 February 2026 – Service monitoring detected failure rates exceeding failure limit thresholds. • 20:10 UTC on 02 February 2026 – We began collaborating to devise a mitigation solution and investigate the underlying factors. • 21:15 UTC on 02 February 2026 – We applied a primary proposed mitigation and validated that it was successful on a test instance. • 21:18 UTC on 02 February 2026 – We identified and disabled the remediation workflow and stopped any ongoing activity, so it would not impact additional storage accounts. • 21:50 UTC on 02 February 2026 – Began broader mitigation to impacted storage accounts. Customers saw improvements as work progressed. • 00:07 UTC on 03 February 2026 – Storage accounts for a few high volume VM extensions that utilize managed identity finished their re-enablement in East US. • 00:08 UTC on 03 February 2026 – Unexpectedly, customer impact increased first in East US and cascaded to West US with retries, as a critical managed identity service degraded under recovery load. • 00:14 UTC on 03 February 2026 – Automated alerting identified availability impact to Managed Identity services in East US. Engineers quickly recognized that the service was overloaded and began to scale out. • 00:30 UTC on 03 February 2026 – All extension hosting storage accounts had been reenabled, mitigating this impact in all regions other than East US and West US. • 00:50 UTC on 03 February 2026 – Initial scale outs of managed identity service infrastructure scale out completed, but the new resources were still unable to handle the traffic volume due to the increasing backlog of retried requests. • 02:00 UTC on 03 February 2026 – A second, larger set of managed identity service scale outs completed. Once again, the capacity was unable to handle the volume of backlogs and retries. • 02:15 UTC on 03 February 2026 – Reviewed additional data and monitored downstream services to ensure that all mitigations were in place for all impacted storage accounts. • 03:55 UTC on 03 February 2026 – To recover infrastructure capacity for the managed identity service, we began rolling out a change to remove all traffic so that the infrastructure could be repaired without load. • 04:25 UTC on 03 February 2026 – After infrastructure nodes recovered, we began gradually ramping traffic to them, allowing backlogged identity operations to begin to process safely. • 06:05 UTC on 03 February 2026 – Backlogged operations completed, and services returned to normal operating levels. We concluded our monitoring and confirmed that all customer impact had been mitigated. How are we making incidents like this less likely or less impactful? • We are strengthening traffic controls and throttling so that unexpected load spikes or retry storms are contained early, and cannot overwhelm the managed identity service or impact other customers. (Estimated completion: February 2026) • We are working to fix the data synchronization problem, by addressing detection gaps and code defects with validation coverage, to protect from similar scenarios. (Estimated completion: March 2026) • We will exercise and optimize the rollback of the remediation workflow, including automation where appropriate, to mitigate issues more quickly. (Estimated completion: April 2026) • We are increasing capacity headroom and tightening scaling safeguards, to ensure sufficient resources remain available during recovery events and sudden demand. (Estimated completion: April 2026) • We will improve orchestration of the remediation workflow to be more in line with our Safe Deployment Practices, integrated with service health indicators, to reduce multi-region failure risk. (Estimated completion: May 2026) • We are improving retry behavior and internal efficiency, so the service degrades more gracefully under stress. (Estimated completion: June 2026) • We are improving regional isolation and failover behavior, to prevent issues in one region from cascading into paired regions during failures or recovery. (Estimated completion: July 2026) How can customers make incidents like this less impactful? • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. • Note that the impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/FNJ8-VQZ

Read the full incident report →

Started Dec 22, 2025, 08:05 AM UTC· resolved Dec 22, 2025, 06:30 PM UTC

1. postmortem Dec 22, 2025, 06:30 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/FV31-PQG What happened? Between 08:05 UTC and 18:30 UTC on 22 December 2025, the Microsoft Entra Privileged Identity Management (PIM) service experienced issues that impacted role activations for a subset of customers. Some requests returned server errors (5xx) and, for a portion of customers, some requests returned unexpected client errors (4xx). This issue manifested as API failures, elevated latency, and various activation errors. Impacted actions could have included: • Reading eligible or active role assignments • Eligible or Active Role assignment management • PIM Policy and PIM Alerts management • Activation of Eligible or Active Role assignments • Deactivation of role assignments initiated by customer (deactivations triggered by the expiration of previous activation were not impacted) • Approval of role assignment activation or extension Customer initiated PIM operations from various Microsoft management portals, mobile app, and API calls were likely impacted. During this incident period, operations that were retried may have succeeded. What went wrong and why? Under normal conditions, Privileged Identity Management processes requests—such as role activations—by coordinating activity across its front-end APIs, traffic routing layer, and the backend databases that store role activation information. These components work together to route requests to healthy endpoints, complete required checks, and process activations without delay. To support this flow, the system maintains a pool of active connections for responsiveness and automatically retries brief interruptions to keep error rates low, helping ensure customers experience reliable access when reading, managing, or activating role assignments. As part of ongoing work to improve how the PIM service stores and manages data, configuration changes which manage the backend databases were deployed incrementally using safe deployment practices. The rollout progressed successfully through multiple rings, with no detected errors and all monitored signals remaining within healthy thresholds. When the deployment reached a ring operating under higher workload, the additional per-request processing increased demand on the underlying infrastructure that hosts the API which manages connections to the backend databases. Although the service includes throttling mechanisms designed to protect against spikes in API traffic, this scenario led to elevated CPU utilization without exceeding request-count thresholds, so throttling did not engage. Over time, the sustained load caused available database connections to be exhausted, and the service became unable to process new requests efficiently. This resulted in delays, timeouts, and errors for customers attempting to view or activate privileged roles. How did we respond? • 21:36 UTC on 15 December 2025 – The configuration change deployment was initiated. • 22:08 UTC on 19 December 2025 – The configuration change was progressed to the ring with the heavy workload. • 08:05 UTC on 22 December 2025 – Initial customer impact began. • 08:26 UTC on 22 December 2025 – Automated alerts received for intermittent, low volume of errors prompting us to start investigating. • 10:30 UTC on 22 December 2025 – We attempted isolated restarts on impacted database instances in an effort to mitigate low-level impact. • 13:03 UTC on 22 December 2025 – Automated monitoring alerted us to elevated error rates and a full incident response was initiated. • 13:22 UTC on 22 December 2025 – We identified that calls to the database were intermittently timing out. Traffic volume appeared to be normal with no significant surge detected however we observed the spike in CPU utilization. • 13:54 UTC on 22 December 2025 – Mitigation efforts began, including beginning to scale out the impacted environments. • 15:05 UTC on 22 December 2025 – Scale out efforts were observed as decreasing error rates but not completely eliminating failures. Further instance restarts provided temporary relief. • 15:25 UTC on 22 December 2025 – Scaling efforts continued. We engaged our database engineering team to help investigate. • 16:37 UTC on 22 December 2025 – While we did not correlate the deployment to this incident, we initiated a rollback of the configuration change. • 17:20 UTC on 22 December 2025 – Scale-out efforts completed. • 17:45 UTC on 22 December 2025 – Service availability telemetry was showing improvements. Some customers began to report recovery. • 18:30 UTC on 22 December 2025 – Customer impact confirmed as mitigated, after rollback of configuration change had completed and error rates had returned to normal levels. How are we making incidents like this less likely or less impactful? • We have rolled back the problematic configuration change across all regions. (Completed) • For outages that manifest later as a result of configuration updates, we are developing a mechanism to help engineers correlate these signals more quickly. (Estimated completion: January 2026) • We are working to ensure this configuration change will not inadvertently introduce excessive load before we redeploy this again. (Estimated completion: January 2026) • We are working on updating our auto-scale configuration to be more responsive to changes in CPU usage. (Estimated completion: January 2026) • We are enabling monitoring and runbooks for available database connections to respond to emerging issues sooner. (Estimated completion: February 2026) How can customers make incidents like this less impactful? • Consider retrying critical operations directly through the API, for example, when requesting just in time access: https://learn.microsoft.com/entra/id-governance/privileged-identity-management/pim-apis#iteration-3-current--pim-for-microsoft-entra-roles-groups-in-microsoft-graph-api-and-for-azure-resources-in-azure-resource-manager-api • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/FV31-PQG

Read the full incident report →

Started Dec 08, 2025, 12:00 AM UTC· resolved Dec 08, 2025, 12:00 AM UTC

1. postmortem Dec 08, 2025, 12:00 AM UTC

   What happened? Between 11:04 and 14:13 EST on 08 December 2025, customers using any of the Azure Government regions may have experienced failures when attempting to perform service management operations through Azure Resource Manager (ARM). This included operations attempted through the Azure Portal, Azure REST APIs, Azure PowerShell, and Azure CLI. Affected services included but were not limited to: Azure App Service, Azure Backup, Azure Communication Services, Azure Data Factory, Azure Databricks, Azure Functions, Azure Kubernetes Service, Azure Maps, Azure Migrate, Azure NetApp Files, Azure OpenAI Service, Azure Policy (including Machine Configuration), Azure Resource Manager, Azure Search, Azure Service Bus, Azure Site Recovery, Azure Storage, Azure Virtual Desktop, Microsoft Fabric, and Microsoft Power Platform (including AI Builder and Power Automate). What went wrong and why? Azure Resource Manager (ARM) is the gateway for management operations for Azure services. ARM does authorization for these operations based on authorization policies, stored in Cosmos DB accounts that are replicated to all regions. On 08 December 2025, an inadvertent automated key rotation resulted in ARM failures to fetch authorization policies that are needed to evaluate access. As a result, ARM was temporarily unable to communicate with underlying storage resources, causing failures in service-to-service communication and affecting resource management workflows across multiple Azure services. This issue surfaced as authentication failures and 500 Internal Server errors to customers across all clients. Because the content of the Cosmos DB accounts for authorization policies is replicated globally, all regions within the Azure Government cloud were affected. Microsoft services use an internal system to manage keys and secrets, which also makes it easy to perform regular needed maintenance activities, such as rotating secrets. Protecting identities and secrets is a key pillar in our Secure Future Initiative to reduce risk, enhance operational maturity, and proactively prepare for emerging threats to identity infrastructure – by prioritizing secure authentication and robust key management. In this case, our ARM service was using a key in a ‘manual mode’ which means that any key rotations would need to be manually coordinated, so that traffic could be moved to use a different key before the key could be regenerated. The Cosmos DBs that ARM use for accessing authorization policies was intentionally onboarded to the Microsoft internal service which governs the account key lifecycle, but unintentionally configured with the option to automatically rotate the keys enabled. This automated rotation should have been disabled as part of the onboarding process, until such time as it was ready to be fully automated. Earlier on the same day of this incident, a similar key rotation issue affected services in the Azure in China sovereign cloud. Both the Azure Government and Azure in China sovereign cloud had their (separate but equivalent) keys created on the same day, starting completely independent timers, back in February 2025 – so each was inadvertently rotated on their respective timers, approximately three hours apart. As such, the key used by ARM for the Azure Government regions was automatically rotated, before the same key rotation issue affecting the Azure in China regions was fully mitigated. Although potential impact to other sovereign clouds was discussed as part of the initial investigation, we did not have a sufficient understanding of the inadvertent key rotation to be able to prevent impact in the second sovereign cloud, Azure Government. How did we respond? • 11:04 EST on 08 December 2025 – Customer impact began. • 11:07 EST on 08 December 2025 – Engineering was engaged to investigate based on automated alerts. • 11:38 EST on 08 December 2025 – We began applying a fix for the impacted authentication components. • 13:58 EST on 08 December 2025 – We began to restart ARM instances, to speed up the mitigation process. • 14:13 EST on 08 December 2025 – All customer impact confirmed as mitigated. How are we making incidents like this less likely or less impactful? • First and foremost, our ARM team have conducted an audit to ensure that there are no other manual keys that are misconfigured to be auto-rotated, across all clouds. (Completed) • Our internal secret management system has paused automated key rotations for managed keys, until usage signals are made available on key usage – see the Cosmos DB change safety repair item below. (Completed) • We will complete the migration to auto-rotated Cosmos DB account keys for ARM authentication accounts, across all clouds. (Estimated completion: February 2026) • Our Cosmos DB team will introduce change safety controls that block regenerating keys that have usage, by emitting a relevant usage signal. (Estimated completion: Public Preview by April 2026, General Availability to follow by August 2026) How can customers make incidents like this less impactful? • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/ML7_-DWG

Read the full incident report →

Started Dec 08, 2025, 12:00 AM UTC· resolved Dec 08, 2025, 12:00 AM UTC

1. postmortem Dec 08, 2025, 12:00 AM UTC

   What happened? Between 16:50 CST on 08 December 2025 and 02:00 CST on 09 December 2025 (China Standard Time) customers using any of the Azure in China regions may have experienced failures when attempting to perform service management operations through Azure Resource Manager (ARM). This included operations attempted through the Azure Portal, Azure REST APIs, Azure PowerShell, and Azure CLI. Affected services included but were not limited to: Azure AI Search, Azure API Management, Azure App Service, Azure Application Insights, Azure Arc, Azure Automation, Azure Backup, Azure Data Factory, Azure Databricks, Azure Database for PostgreSQL Flexible Server, Azure Kubernetes Service, Azure Logic Apps, Azure Managed HSM, Azure Marketplace, Azure Monitor, Azure Policy (including Machine Configuration), Azure Portal, Azure Resource Manager, Azure Site Recovery, Azure Stack HCI, Azure Stream Analytics, Azure Synapse Analytics, and Microsoft Sentinel. What went wrong and why? Azure Resource Manager (ARM) is the gateway for management operations for Azure services. ARM does authorization for these operations based on authorization policies, stored in Cosmos DB accounts that are replicated to all regions. On 08 December 2025, an inadvertent automated key rotation resulted in ARM failures to fetch authorization policies that are needed to evaluate access. As a result, ARM was temporarily unable to communicate with underlying storage resources, causing failures in service-to-service communication and affecting resource management workflows across multiple Azure services. This issue surfaced as authentication failures and 500 Internal Server errors to customers across all clients. Because the content of the Cosmos DB accounts for authorization policies is replicated globally, all regions within the Azure in China cloud were affected. Azure services use an internal system to manage keys and secrets, which also makes it easy to perform regular needed maintenance activities, such as rotating secrets. Protecting identities and secrets is a key pillar in our Secure Future Initiative to reduce risk, enhance operational maturity, and proactively prepare for emerging threats to identity infrastructure – by prioritizing secure authentication and robust key management. In this case, our ARM service was using a key in a ‘manual mode’ which means that any key rotations would need to be manually coordinated, so that traffic could be moved to use a different key before the key could be regenerated. The Cosmos DBs that ARM use for accessing authorization policies was intentionally onboarded to the internal service which governs the account key lifecycle, but unintentionally configured with the option to automatically rotate the keys enabled. This automated rotation should have been disabled as part of the onboarding process, until such time as it was ready to be fully automated. How did we respond? • 16:50 CST on 08 December 2025 – Customer impact began. • 16:59 CST on 08 December 2025 – Engineering was engaged to investigate based on automated alerts. • 18:37 CST on 08 December 2025 – We identified the underlying cause as the incorrect key rotation. • 19:16 CST on 08 December 2025 – We identified mitigation steps and began applying a fix for the impacted authentication components. This was tested and validated before being applied. • 22:00 CST on 08 December 2025 – We began to restart ARM instances, to speed up the mitigation process. • 23:53 CST on 08 December 2025 – Many services had recovered but residual impact remained for some services. • 02:00 CST on 09 December 2025 – All customer impact confirmed as mitigated. How are we making incidents like this less likely or less impactful? • First and foremost, our ARM team have conducted an audit to ensure that there are no other manual keys that are misconfigured to be auto-rotated, across all clouds. (Completed) • Our internal secret management system has paused automated key rotations for managed keys, until usage signals are made available on key usage – see the Cosmos DB change safety repair item below. (Completed) • We will complete the migration to auto-rotated Cosmos DB account keys for ARM authentication accounts, across all clouds. (Estimated completion: February 2026) • Our Cosmos DB team will introduce change safety controls that block regenerating keys that have usage, by emitting a relevant usage signal. (Estimated completion: Public Preview by April 2026, General Availability to follow by August 2026) How can customers make incidents like this less impactful? • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/JSNV-FBZ

Read the full incident report →

Started Nov 05, 2025, 04:53 PM UTC· resolved Nov 06, 2025, 02:25 AM UTC

1. postmortem Nov 06, 2025, 02:25 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/2LGD-9VG What happened? Between approximately 16:53 UTC on 5 November 2025 and 02:25 UTC on 6 November 2025, a subset of customers in the West Europe region experienced service disruptions or degraded performance across multiple services, including Virtual Machines (VM), Azure Database for PostgreSQL Flexible Server, MySQL Flexible Server, Azure Kubernetes Service, Storage, Service Bus, and Virtual Machine Scale Sets, among others. Customers using Azure Databricks in West Europe observed degraded performance when launching or scaling all-purpose and job compute workloads, which impacted Unity Catalog and Databricks SQL operations. What went wrong and why? The incident was triggered by a voltage sag in the utility grid, which caused cooling units to shut down and temperatures to rise above normal thresholds in a datacenter located in Physical Availability Zone 01 in the West Europe region. As temperature rose, multiple storage scale units automatically powered down to prevent hardware damage. Under normal circumstances, cooling systems are designed and commissioned to automatically restart after power events and are built with sufficient redundancy to maintain safe operating conditions even when failures occur. While cooling unit auto-restart capability had been regularly tested and exercised in this datacenter, the specific size and duration of utility sag exposed a previously unknown hardware issue that prevented the cooling units from restarting. This resulted in temperatures exceeding safe operational limits for infrastructure. The extended recovery time was influenced by several factors. Recovery of storage scale units was prolonged because some storage servers entered a degraded state, necessitating sequential validation before they could be brought back online. In addition, dependent services such as compute and networking could not resume until storage integrity checks were complete. Finally, additional thermal audits were performed before workloads were reactivated to ensure that no residual risk remained. How did we respond? Automated monitoring detected the temperature anomaly and triggered an incident response. Facilities teams restored safe operating conditions by performing hard reset on the affected units and confirmed that temperatures were trending back to safe levels. Service recovery efforts began once the facility had reached safe operating temperatures. Following this, storage systems initiated comprehensive data consistency checks when nodes in a storage scale unit restarted after the unplanned shutdown. These checks, which may include reconstructing replicas that have fallen behind. The recovery process prioritizes data integrity and durability over availability, resulting in customer traffic being blocked until all integrity validations was completed and contributing to the extended recovery time. In parallel, engineers prioritized recovery of compute hosts to stop further VM failures and allow dependent services to resume. Timeline of events: • 16:53 UTC on 5 November 2025 – Customer impact began as elevated temperatures caused multiple storage scale units to shut down. • 16:55 UTC on 5 November 2025 – Datacenter monitoring detected temperature breaches and triggered thermal alerts. • 17:20 UTC on 5 November 2025 – Power sag and cooling unit failure to restart identified as contributing factors. Cooling recovery begins as engineers manually restart cooling units. • 17:40 UTC on 5 November 2025 – Mitigation workstream started as cooling restoration progresses. • 17:48 UTC on 5 November 2025 – All cooling units restored, we start seeing temperatures reducing as cooling normalized. • 17:50 UTC on 5 November 2025 – Rack-level thermal monitoring returned to safe operational thresholds. Engineering teams initiated storage recovery. • 18:30 UTC on 5 November 2025 – Sequential storage scale unit validation began. Each scale unit underwent integrity checks to ensure no data corruption before bringing storage services back into rotation. • 20:00 UTC on 5 November 2025 – Gradual restoration of storage scale units. Recovery was staged to avoid overloading power and cooling systems. Dependent compute nodes remained offline during this time. • 23:30 UTC on 5 November 2025 – Service dependency checks and orchestration. Networking and compute services were progressively re-enabled after storage scale units passed health checks. • 02:25 UTC on 6 November 2025 – All services were brought back online, and customer impact was mitigated. How are we making incidents like this less likely or less impactful? We are implementing several improvements to strengthen resilience and accelerate recovery: • We are conducting internal and manufacturer-led forensic investigations of specific device hardware and firmware to develop solutions to bolster component tolerance to power sags and voltage fluctuations. (Estimated completion: December 2025) • We are performing a gap analysis of the control circuit to ensure all relays delivering critical function of the units are powered through Uninterruptible Power Supply (UPS) and stored energy sources. (Estimated completion: January 2026) • We are validating our maintenance procedures and adding additional prechecks for Heating, Ventilation, and Air Conditioning (HVAC) subsystems. (Estimated completion: January 2026) • We are optimizing recovery steps for compute and storage systems to reduce overall restoration time. (Estimated completion: January 2026) • We are improving automated correlation between environmental signals and service health to trigger earlier, and more targeted escalations. (Estimated completion: January 2026) • We are updating our cooling unit commissioning playbooks to expand range of power sag and swell scenarios. (Estimated completion: February 2026) • In the longer term, we are implementing a new approach to accelerate service restoration after major incidents. Historically, recovery prioritized full data integrity checks before bringing services online, which extended downtime because it was a serial process. Going forward, integrity checks will run in the background during VM and disk mount operations, allowing availability to return much sooner without compromising data safety. This change is a direct outcome of this incident and represents a significant improvement in recovery speed. (Estimated completion: April 2026) How can customers make incidents like this less impactful? • Consider using Availability Zones (AZs) to run your services across physically separate locations within an Azure region. To help services be more resilient to datacenter-level failures like this one, each AZ provides independent power, networking, and cooling. Many Azure services support zonal, zone-redundant, and/or always-available configurations: https://docs.microsoft.com/azure/availability-zones/az-overview • Plan for regional redundancy by using active/active or failover designs across paired regions. • Maintain regular backups and test failover procedures to ensure recovery readiness. • Increase resilience by using retry logic and backoff strategies to prevent overload during recovery. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/2LGD-9VG

Read the full incident report →

Started Oct 29, 2025, 03:41 PM UTC· resolved Oct 30, 2025, 12:05 AM UTC

1. postmortem Oct 30, 2025, 12:05 AM UTC

   Read Azure Front Door's blog series in making progress on key resilience repairs: https://aka.ms/AzureFrontDoor/Resiliency-Part1 Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/YKYN-BWZ What happened? Between 15:41 UTC on 29 October and 00:05 UTC on 30 October 2025, customers and Microsoft services leveraging Azure Front Door (AFD) and Azure Content Delivery Network (CDN) experienced connection timeout errors and Domain Name System (DNS) resolution issues. From 18:30 UTC on 29 October 2025, as the system recovered gradually, some customers started to see availability improve – albeit with increased latency – until the system fully stabilized by 00:05 UTC on 30 October 2025. Affected Azure services included, but were not limited to: Azure Active Directory B2C, Azure AI Video Indexer, Azure App Service, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Marketplace, Azure Media Services, Azure Portal, Azure Sphere Security Service, Azure SQL Database, and Azure Static Web Apps. Other Microsoft services were also impacted, including Microsoft 365 (see: MO1181369), the Microsoft Communication Registry website, Microsoft Copilot for Security, Microsoft Defender (External Attack Surface Management), Microsoft Dragon Copilot, Microsoft Dynamics 365 and Power Platform (see: MX1181378), Microsoft Entra ID (Mobility Management Policy Service, Identity & Access Management, and User Management), Microsoft Purview, Microsoft Sentinel (Threat Intelligence), Visual Studio App Center, and customers’ ability to open support cases (both in the Azure Portal and by phone). What went wrong and why? Azure Front Door (AFD) and Azure Content Delivery Network (CDN) route traffic using globally distributed edge sites supporting customers as well as Microsoft services including various management portals. The AFD control plane generates customer configuration metadata that the data plane consumes for all customer-initiated operations including purge and Web Application Firewall (WAF) on the AFD platform. Since customer applications hosted on AFD and CDN can be accessed by their end users from anywhere in the world, these changes are deployed globally across all its edge sites to provide a consistent user experience. A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service. This defect escaped detection due to a gap in our pre-production validation, since not all features are validated across different control plane build versions. Azure Front Door employs multiple deployment stages, and a configuration protection system to ensure safe propagation of customer configurations. This system validates configurations at each deployment stage and advances only after receiving positive health signals from the data plane. Once deployments are rolled out successfully, the configuration propagation system also updates a ‘Last Known Good’ (LKG) snapshot (a periodic snapshot of healthy customer configurations) so that deployments can be automatically rolled back in case of any issues. The configuration protection system waits for approximately a minute between each stage, completing on an average within 5-10 minutes globally. During this incident, the incompatible customer configuration change was made at 15:35 UTC, and was applied to the data plane in a pre-production stage at 15:36 UTC. Our configuration propagation monitoring continued to receive healthy signals – although the problematic metadata was present, it had not caused any issues. Because the data plane crash surfaced asynchronously, after approximately five minutes, the configuration passed through the protection safeguards and propagated to later stages. This configuration (with the incompatible metadata) completed propagation to a majority of edge sites by 15:39 UTC. Since the incompatible customer configuration metadata was deployed successfully to the majority of fleet with positive health signal, the LKG was also updated with this configuration. The data plane impact began in phases starting with our preproduction environment at 15:41 UTC, and replicated across all edge sites globally by 15:45 UTC. As the data plane impact started, the configuration protection system detected this and stopped all new and inflight customer configuration changes from being propagated at 15:43 UTC. The incompatible customer configuration was processed by edge servers, causing crashes across our various edge sites. This also impacted AFD’s internal DNS service, hosted on the edge sites of Azure Front Door, resulting in intermittent DNS resolution errors for a subset of AFD customer requests. This sequence of events was the trigger for the global impact on the AFD platform. This AFD incident on 29 October was not directly related to the previous AFD incident, from 9 October. Both incidents were broadly related to configuration propagation risk (inherent to a global Content Delivery Network, in which route/WAF/origin changes must be quickly deployed worldwide) but while the failure mode was similar, the underlying defects were different. Azure Front Door’s configuration protection system is designed to validate configurations and proceed only after receiving positive health signals from the data plane. During the AFD incident on 9 October (Tracking ID: QNBQ-5W8) that protection system worked as intended, but was later bypassed by our engineering team during a manual cleanup operation. During this AFD incident on 29 October (Tracking ID: YKYN-BWZ) the incompatible customer configuration metadata progressed through the protection system, before the delayed asynchronous processing task resulted in the crash. Some of the learnings and repair items from the earlier incident are applicable to this incident as well, and are included in the list of repairs below. How did we respond? The issue started at 15:41 UTC and was detected by monitoring at 15:48 UTC, prompting our investigation. By 15:43 UTC the configuration protection system activated in response to widespread data plane issues, and automatically blocked all new and in-flight configuration changes from being deployed worldwide. Since the latest ‘last known good’ (LKG) version was updated with the conflicting metadata, we chose not to revert to it. To ensure system stability, we decided not to rollback to prior versions of the LKG either. Instead, we opted to edit the latest LKG, by removing the problematic customer configurations manually. We also opted to block all customer configuration changes from propagating to the data plane at 17:30 UTC so that, as we mitigate, we would not reintroduce this issue. At 17:40 UTC we began deploying the updated LKG configuration across the global fleet. Recovery required reloading all customer configurations at every edge site and rebalancing traffic gradually, to avoid overload conditions as Edge sites returned to service. This deliberate, phased recovery was necessary to stabilize the system while restoring scale and ensuring no recurrence of the issue. Many downstream services that use AFD were able to failover to prevent further customer impact, including Microsoft Entra and Intune portals, and Azure Active Directory B2C. In a more complex example, Azure Portal leveraged its standard recovery process to successfully transition away from AFD during the incident. Users of the Portal would have seen limited impact during this failover process and then been able to use the Portal without issue. Unfortunately, some services within the Portal did not have an established fallback strategy and therefore parts of the Portal experience continued to experience failures even after Portal recovery (for example, Marketplace). Timeline of major incident milestones: • 15:35 UTC on 29 October 2025 – Corrupt metadata first introduced, as described above. • 15:41 UTC on 29 October 2025 – Customer impact began, triggered by the resulting crashes. • 15:43 UTC on 29 October 2025 – Configuration protection system activated in response to issues. • 15:48 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. • 16:15 UTC on 29 October 2025 – Focus of the investigation became examining AFD configuration changes. • 16:18 UTC on 29 October 2025 – Initial communication posted to our public status page. • 16:20 UTC on 29 October 2025 – Targeted communications to impacted customers sent to Azure Service Health. • 17:10 UTC on 29 October 2025 – Began updating the ‘last known good’ LKG configuration to remove problematic configurations manually. • 17:26 UTC on 29 October 2025 – Azure Portal failed away from Azure Front Door. • 17:30 UTC on 29 October 2025 – Blocked all customer configuration propagation to the data plane, in preparation for deploying the new configuration. • 17:40 UTC on 29 October 2025 – Initiated the deployment of our updated ‘last known good’ configuration. • 17:50 UTC on 29 October 2025 – Last known good configuration available to all Edge sites, which began gradually reloading LKG configuration. • 18:30 UTC on 29 October 2025 – AFD DNS servers recovered, allowing us to rebalance traffic manually to a small number of healthy Edge sites. Customers began seeing improvements in availability. • 20:20 UTC on 29 October 2025 – As a sufficient number of Edge sites had recovered, we switched to automatic traffic management – customers continued to see availability improve. • 00:05 UTC on 30 October 2025 – AFD impact confirmed mitigated for customers, as availability and latency had returned to pre-incident levels. Post mitigation, we temporarily blocked all AFD customer configuration changes at the Azure Resource Manager (ARM) level to ensure the safety of the data plane. We also implemented additional safeguards including (i) fixing the control plane and data plane defects, (ii) removing asynchronous processing from the data plane, (iii) introducing an additional ‘pre-canary’ stage to test customer configuration (iv) extending the bake time during each stage of the configuration propagation, and (v) improvements to the data plane recovery time from approximately 4.5 hours to approximately one hour. We began draining the customer configuration queue from 2 November 2025. Once these safeguards were fully implemented, this restriction was removed on 5 November 2025. The introduction of new stages in the configuration propagation pipeline was coupled with additional ‘bake time’ between stages – which has resulted in an increase in configuration propagation time, for all operations including create, update, delete, WAF operations on AFD platform, and cache purges. We continue to work on platform enhancements to ensure a robust configuration delivery pipeline and further reduce the propagation time. For more details on these temporary propagation delays, refer to http://aka.ms/AFD_FAQ. How are we making incidents like this less likely or less impactful? To prevent issues like this, and improve deployment safety... • We have fixed both the original control plane incompatibility, and the data plane bug described above. (Completed) • We are now enforcing complete synchronous processing of each customer configuration, before advancing to production stages. (Completed) • We have implemented additional stages in our phased configuration rollout, including extended bake time to help detect configuration related issues. (Completed) • We are decoupling configuration processing in data plane servers from active traffic-serving instances to isolated worker process instances, thereby removing the risk of any configuration defect impacting data plane processing. (Estimated completion: January 2026) • Once our pre-validation of this configuration pipeline is in place, we will work towards reducing the propagation time from 45 minutes to approximately 15 minutes. (Estimated completion: January 2026) • We are enhancing our testing and validation framework to ensure backwards compatibility with configurations generated across previous versions of the control plane build. (Estimated completion: February 2026) To reduce the blast radius of potential future issues... • We have migrated critical first-party infrastructure (including Azure Portal, Azure Communication Services, Marketplace, Linux Software Repository for Microsoft Products, Support ticket creation) into an active-active solution with fail away. (Completed) • In the longer term, we are enhancing our customer configuration and traffic isolation, to ensure that no impact to any other customers from single customers’ traffic or configuration issue – utilizing ‘micro cell’ segmentation of the AFD data plane. (Estimated completion: June 2026) To be able to recover more quickly from issues... • We have made changes to accelerate our data plane recovery time, by leveraging the local customer configuration caching more effectively – to restore customer configurations within one hour. (Completed) • We are making investments to reduce data plane recovery time further, to restore customer configurations within approximately 10 minutes. (Estimated completion: March 2026) To improve our communications and support... • We have addressed delays in delivering alerts via Azure Service Health to impacted customers, by making immediate improvements to increase resource thresholds. (Completed) • We will expand the automated customer alerts sent via Azure Service Health, to include similar classes of service degradation – to notify impacted customers more quickly. (Estimated completion: November 2025) • Finally, we will resolve the technical and staffing challenges that prevented Premier and Unified customers from being able to create a support request, by ensuring that we can failover to our backups systems more quickly. (Estimated completion: November 2025) How can customers make incidents like this less impactful? • Understand best practices for building a resilient global HTTP ingress layer, to maintain availability during regional or network disruptions: https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/mission-critical-global-http-ingress • As an alternative for when management portals are temporarily inaccessible, customers can consider using programmatic methods to manage resources – including the REST API (https://learn.microsoft.com/rest/api/azure) and PowerShell (https://learn.microsoft.com/powershell/azure) • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/YKYN-BWZ

Read the full incident report →

Started Oct 09, 2025, 07:43 PM UTC· resolved Oct 09, 2025, 11:59 PM UTC

1. postmortem Oct 09, 2025, 11:59 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/QKNQ-PB8 What happened? Between 19:43 UTC and 23:59 UTC on 09 October 2025, customers may have experienced availability issues and failures when loading content for the Azure Portal as well as other management portals across Microsoft. Over the course of the incident, approximately 45% of customers using the management portals experienced some form of impact – with failure rates reaching their peak at approximately 20:54 UTC, after which time customers generally experienced availability improving when attempting to load content. Note that service management via programmatic methods (such as PowerShell or the REST API) and the availability of resources were not impacted. What went wrong and why? Various Microsoft management portals utilize backend services to host content for different portal extensions. Azure Front Door (AFD) and Content Delivery Network (CDN) services are used to accelerate delivery of application content. An earlier incident occurred (Tracking ID: QNBQ-5W8) impacting the availability of management portals primarily from Africa and Europe, with lesser impact to Asia Pacific and the Middle East. We invoked our Business Continuity Disaster Recovery (BCDR) protocol and updated our network filters to allow traffic to bypass AFD so that customer traffic could reach backend services hosting content. Automation scripts were used to update the traffic load balancing configuration to split traffic across multiple routes. However, the scripts inadvertently removed a configuration value, because the script used an API version that predated the introduction of that value. The AFD endpoint began failing its health checks because the monitor of the AFD route falsely interpreted it as unhealthy. As a result, it was no longer routing traffic. However, since traffic was routed through alternative paths, there was no impact to customers, and we were not aware that the configuration value was removed. These automation scripts had previously been used successfully in other scenarios. After recovery of the previous incident, we began taking steps to resume normal traffic through AFD by using these automation scripts. However, due to the previously mentioned configuration value, traffic was not served correctly. We updated network filters to allow customer traffic only through AFD and, as this propagated, the backend hosting services were no longer reachable via AFD. This resulted in the impact to customers starting at 19:43 UTC. How did we respond? • 11:59 UTC on 09 October 2025 – We performed operations to allow management portal traffic to bypass AFD. • 19:39 UTC on 09 October 2025 – After verifying recovery of the previous incident, we took steps to migrate management portal traffic back completely through AFD. • 19:43 UTC on 09 October 2025 – Management portal availability issues occurred and impact grew. • 19:44 UTC on 09 October 2025 – Initial automated alerts were triggered. • 19:50 UTC on 09 October 2025 – Engineers were engaged and began investigating. • 20:30 UTC on 09 October 2025 – We confirmed the issues were related to the traffic migration and continued to investigate contributing factors. • 20:54 UTC on 09 October 2025 – We recovered one hosting service domain to correctly resume traffic through AFD and purged its caches to speed up recovery. Gradual recovery happened over the next 60 minutes, though other residual impact continued. • 21:30 UTC on 09 October 2025 – Azure Portal availability was mitigated, however other residual impact remained for other management portals. We shifted our focus to investigating these other residual failures and mitigating this impact. • 23:59 UTC on 09 October 2025 – We recovered an additional hosting service domain to correctly resume traffic through AFD. After reviewing our configuration and monitoring telemetry, we confirmed mitigation. How are we making incidents like this less likely or less impactful? • We have audited our architecture for the traffic routing of our management portals. (Completed) • We have updated our AFD migration scripts to use the latest API versions, to ensure that future runs will not remove the problematic configuration value. (Completed) • We are updating our standard operating procedures to add additional validation steps, to help ensure confirmation that traffic is being routed as expected. (Estimated completion: October 2025) • We are hardening our client-side logic so that browsers can fallback from AFD and hosting service incidents. (Estimated completion: November 2025) • We are making improvements to our failover systems from AFD, to be more robust and automated, so no manual intervention is required. (Estimated completion: December 2025) • We will be conducting additional simulation drills on our internal environments, to ensure that traffic shifts exactly as expected. (Estimated completion: January 2026) • In the longer term, we will be updating the architecture of our infrastructure to support regional rollout of such network traffic changes, to reduce the potential impact radius. (Estimated completion: March 2026) How can customers make incidents like this less impactful? • As an alternative for when management portals are inaccessible, customers can consider using programmatic methods to manage resources – including the REST API (https://learn.microsoft.com/rest/api/azure) and PowerShell (https://learn.microsoft.com/powershell/azure) • During service incidents in which Azure Portal content is unavailable, customers can try using our preview Azure Portal endpoint URL: https://preview.portal.azure.com • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/QKNQ-PB8

Read the full incident report →

Started Oct 09, 2025, 07:50 AM UTC· resolved Oct 09, 2025, 04:00 PM UTC

1. postmortem Oct 09, 2025, 04:00 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/QNBQ-5W8 What happened? Between 07:50 UTC and 16:00 UTC on 09 October 2025, Microsoft services and Azure customers leveraging Azure Front Door (AFD) and Azure Content Delivery Network (CDN) may have experienced increased latency and/or timeouts – primarily across Africa and Europe, as well as Asia Pacific and the Middle East. This impacted the availability of the Azure Portal as well as other management portals across Microsoft. Peak failure rates for AFD reached approximately 17% in Africa, 6% in Europe, and 2.7% in Asia Pacific and the Middle East. Availability was restored by 12:50 UTC, though some customers continued to experience elevated latency. Latency returned to baseline levels by 16:00 UTC, at which point the incident was mitigated. What went wrong and why? AFD routes traffic using globally distributed edge sites and supports Microsoft services including the management portals. The AFD control plane generates system metadata that the data plane consumes for customer-initiated ‘create’, ‘update’, or ‘delete’ operations on AFD or CDN profiles. One of the trigger conditions for this incident was a software defect in the latest version of the AFD control plane which had been rolled out six weeks prior to the incident, in line with our safe deployment practices. Newly created customer tenant profiles were being onboarded to the newer control plane version. Our service monitoring detected elevated data plane crashes due to a previously unknown bug – triggered by erroneous metadata, generated by a particular sequence of profile update operations. Our automated protection layer intercepted this in early update stages and prevented this metadata from propagating any further to the data plane, thereby averting any customer impact at that time. In addition, as the newer control plane was running in tandem with the previous version of the control plane, we disabled the new control plane from taking any requests. On 09 October 2025, we initiated a cleanup of the affected tenant configuration with the erroneous metadata. Since the automated protection system was blocking the impacted customer tenant profile updates in the initial stage, we temporarily bypassed it to allow the cleanup of the tenant configuration to proceed. By bypassing the protection system, the erroneous metadata was inadvertently able to propagate to later stages – and triggered the bug in the data plane that crashed the data plane service. This resulted in a disruption to a significant number of edge sites across Europe and Africa, approximately 26% of AFD data plane infrastructure resources in these regions were impacted. As part of AFD mechanisms to manage traffic, load was automatically distributed to nearby edge sites (including in Asia Pacific and the Middle East). Additionally, as regional business hours traffic started ramping up, it added to the overall traffic load. The increased volume of traffic on the remaining healthy edge sites resulted in high resource utilization, which exceeded operational thresholds. This triggered an additional layer of protection which started distributing traffic to a broader set of edge sites globally, to reduce further impact. Recovery required a combination of automated restarts, manual intervention where automated restarts were taking too long, and traffic failover operations for impacted management portals. Full mitigation was achieved once edge site infrastructure resources stabilized and latency returned to normal. Additionally, initial customer notifications were delayed primarily due to challenges determining impact, while attempting to target communications to those impacted. We have automated communications to notify customers of incidents quickly, unfortunately this capability was not yet supported in this incident scenario. How did we respond? • 07:30 UTC on 09 October 2025 – The cleanup operation was initiated. • 07:50 UTC on 09 October 2025 – Initial customer impact began, and increased over the next 90 minutes. • 08:13 UTC on 09 October 2025 – Our telemetry detected resource availability loss across multiple AFD edge sites. We began investigating as impact continued to grow. • 09:04 UTC on 09 October 2025 – We identified that the crashes were due to the previously identified data plane bug. • 09:08 UTC on 09 October 2025 – Automated restarts began for our AFD infrastructure resources, and manual intervention began for resources that did not recover automatically. • 09:15 UTC on 09 October 2025 – Customer impact had grown to be at its peak. • 10:01 UTC on 09 October 2025 – Communications were published to the Azure Status page. • 10:45 UTC on 09 October 2025 – Targeted customer communications were sent to Azure Service Health. • 11:59 UTC on 09 October 2025 – Management portals, like the Azure Portal, performed failover operations (including using scripts to update the load balancing configuration, to split traffic between multiple routes) helping restore its service availability. • 12:50 UTC on 09 October 2025 – Availability for AFD fully recovered, however a subset of customers may still have been experiencing elevated latency. • 16:00 UTC on 09 October 2025 – After continuous monitoring of latency improvement, we declared the incident as mitigated after confirming recovery. How are we making incidents like this less likely or less impactful? • We have hardened our standard operating procedures, to ensure that the configuration protection system is not bypassed for any operation. (Completed) • We have fixed the control plane defect which generated the erroneous tenant metadata that led to the data plane resource crashes. (Completed) • We have fixed the bug in the data plane. (Completed) • We will expand the automated customer alerts sent via Azure Service Health, to include similar classes of service degradation. (Estimated completion: November 2025) • We are making improvements to our Azure Portal failover systems from AFD, to be more robust and automated. (Estimated completion: December 2025) • We are building additional runtime configuration validation pipelines against a replica of real-time data plane, as a pre-validation step prior to applying them broadly. (Estimated completion: March 2026) • We are improving data plane resource instance recovery time, following any impact to the data plane. (Estimated completion: March 2026) How can customers make incidents like this less impactful? • Consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins: https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/overview • Consider reviewing our best practices for Azure Front Door architecture: https://learn.microsoft.com/azure/well-architected/service-guides/azure-front-door • Consider implementing retry patterns with exponential backoff, to improve workload resiliency: https://learn.microsoft.com/azure/architecture/patterns/retry • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: http://aka.ms/AzPIR/QNBQ-5W8

Read the full incident report →

Started Sep 26, 2025, 11:54 PM UTC· resolved Sep 27, 2025, 09:59 PM UTC

1. postmortem Sep 27, 2025, 09:59 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/BT6W-FX0 What happened? Between 23:54 UTC on 26 September 2025 and 21:59 UTC on 27 September 2025, a platform issue resulted in an impact to customers in Switzerland North who may have experienced service availability issues for resources hosted in the region. Many customers experienced mitigation by 04:00 UTC on 27 September 2025, while those impacted by additional residual impacts that remained after initial mitigation steps experienced full mitigation as late as 21:59 UTC on 27 September 2025. Impacted services included Azure API Management, Azure App Service, Azure Application Gateway, Azure Application Insights, Azure Cache for Redis, Azure Cosmos DB, Azure Data Explorer, Azure Database for PostgreSQL, Azure Databricks, Azure Firewall, Azure Kubernetes Service, Azure Storage, Azure Synapse Analytics, Azure Backup, Azure Batch, Azure Data Factory, Azure Log Analytics, Azure SQL Database, Azure SQL Managed Instance, Azure Site Recovery, Azure Stream Analytics, Azure Virtual Machine Scale Sets (VMSS), Azure Virtual Machines (VMs), and Microsoft Sentinel. Other services reliant on these may have also been impacted. What went wrong and why? A planned configuration change was made to the certificates used to authorize the communication for our software load balancer infrastructure that routes traffic for services in the Switzerland North region. One of the new certificates contained a malformed value that was not identified during the validation process. Moreover, this specific configuration change went through an expedited path - which unexpectedly applied it across multiple availability zones in the region - without evaluation of system health safeguards. This type of planned change is intended to be impactless but, due to the malformed certificate, authorization of connectivity for load balancer between nodes and resources could not complete. This caused significant impact to storage resources in the region. When a VM loses connectivity to its underlying virtual disk for an extended period, the platform detects the issue and safely shuts down the VM until connection is re-established, to prevent data corruption. Impact magnified because of services dependent on these resources. We corrected the certificate error and deployed the fix to re-establish connectivity for load balancer within the region. Most affected resources automatically recovered once connectivity was restored. However, a subset of resources (such as SQL Managed Instance VMs, VMs using Trusted Launch configurations, some Service Fabric Managed Cluster based services, and services that depend on them) did not resume operation automatically. These remained in a stopped or unhealthy state. • For SQL Managed Instance VMs, after load balancer connectivity was restored, the VMs remained unhealthy. This was due to a race condition during VM startup where different service specific operations customized for this type of VM were not completed in the expected sequence. This required each impacted VM to have its services restarted. • For Trusted Launch VMs, which use Trusted Launch with guest state stored in remote blobs, a unique failure mode was experienced. This incident resulted in a shutdown signal that is also used when customers intentionally reboot their VMs. The platform misinterpreted these events as customer-initiated and did not trigger automated recovery. As a result, these VMs remained in a stopped state and required explicit manual intervention to restart. • For Service Fabric Managed Cluster based services, a specific custom extension is present which required a specific sync pattern to succeed in reaching the storage account in order to provision the VM. For a subset of resources, this timed out due to transient failures. Once the VMs were properly restarted, these recovered. How did we respond? • 23:54 UTC on 26 September 2025 – Customer impact began, triggered by the configuration change. • 00:08 UTC on 27 September 2025 – The issue was detected via our automated monitoring. • 00:12 UTC on 27 September 2025 – Investigation commenced by our Azure Storage and Networking engineering teams. • 02:33 UTC on 27 September 2025 – We performed steps to revert the configuration change for the Switzerland North region. • 03:40 UTC on 27 September 2025 – The certificate was successfully reverted for the region. • 04:00 UTC on 27 September 2025 – Validation of recovery was confirmed for the majority of impacted services, but a subset of resources were still unhealthy due to residual impact. We performed an extended health evaluation to confirm which remaining resources required recovery, and investigated the steps were needed to recover residual impact. • 06:19 UTC on 27 September 2025 – We took additional time to recover remaining residual impact which included developing and validating new scripts to start resources that did not recover automatically. • 11:24 UTC on 27 September 2025 – Performed necessary checks to test safe recovery actions for different types of resources. • 14:15 UTC on 27 September 2025 – Recovery operations were performed for residual impact. This included executing mitigation scripts, and performing steps to safely reboot and recover remaining resources. • 16:15 UTC on 27 September 2025 – Partial recovery for residual impact. Additional operations to recover the remaining impact included properly restarting the resources in a stopped state. • 21:59 UTC on 27 September 2025 – Residual impact recovery activities and validation completed, confirming full recovery of all impacted services and customers. How are we making incidents like this less likely or less impactful? • We have implemented additional auditing into our deployment systems, designed to de-risk issues like this one. (Completed) • We have extensively evaluated our deployment pipelines and removed expedited pipelines that could result in this class of issue. (Completed) • We have improved our deployment safety measures for such changes, with additional automated rollback capabilities at earlier stages. (Completed) • We are improving our monitoring signals, to better understand the health posture of different resources and to expedite recovery. (Estimated completion: November 2025) • In the longer term, we are investing in improvements to our recovery processes for residual impact after these events. In particular, we will apply robust improvements for different custom extension processes and improve the resiliency of different unique resource startup operations to avoid requiring manual intervention. How can customers make incidents like this less impactful? • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones#deployment-approach-4-multi-region-deployments and https://learn.microsoft.com/azure/well-architected/design-guides/disaster-recovery • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/BT6W-FX0

Read the full incident report →

Started Sep 10, 2025, 09:03 AM UTC· resolved Sep 10, 2025, 06:50 PM UTC

1. postmortem Sep 10, 2025, 06:50 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/VKY3-PF8 What happened? Between 09:03 UTC and 18:50 UTC on 10 September 2025, an issue with the Allocator service in the Azure Compute control plane, led to degraded service management operations for resources in two Availability Zones (physical AZ02 and AZ03) in the East US 2 region. While the Allocator issue was mitigated by 18:50 UTC, a subset of downstream services experienced delayed recovery due to backlogged operations and retry mechanisms, with full recovery observed by 19:30 UTC. During this impact window, customers may have received error notifications and/or been unable to start, stop, scale, or deploy resources. The primary services affected were Virtual Machines (VMs) and Virtual Machines Scale Sets (VMSS) but services dependent on them were also affected – including but not limited to: • Azure Backup: Between 13:32 UTC and 19:30 UTC on 10 September 2025, customers may have experienced failures with Virtual Machine backup operations. • Azure Batch: Between 09:50 UTC and 18:50 UTC on 10 September 2025, customers may have experienced pool operations getting stuck. Primarily, pool grow/shrink and deletion operations may have become stuck. The majority of impact occurred between 15:06 and 18:50 UTC, but backlogged operations would have caught up by 19:30 UTC. • Azure Databricks: Between 10:15 UTC and 18:07 UTC on 10 September 2025, customers may have experienced delays and/or failures with job run operations and/or Databricks SQL queries. • Azure Data Factory: Between 10:32 UTC and 18:08 UTC on 10 September 2025, customers may have experienced failures in running dataflow jobs due to cluster creation issues. • Azure Kubernetes Service: Between 09:05 UTC and 19:30 UTC on 10 September 2025, customers may have experienced failures for operations, including but not limited to - create, upgrade, scale up, or delete managed clusters or node pools, or create or delete certain pods which require scaling up nodes, or attach/detach disks to VMs. • Azure Synapse Analytics: Between 10:28 UTC and 18:21 UTC on 10 September 2025, customers may have experienced spark job execution failures while executing spark activities within notebooks, data pipelines, or SDKs. • Microsoft Dev Box: Between 12:30 UTC and 17:15 UTC on 10 September 2025, customers may have experienced operation failures, including manual start via Dev Portal and connecting directly via the Windows App to a Dev Box in a shutdown or hibernated state. While AZ01 was not affected by the allocator service issue that impacted AZ02 and AZ03, allocation failures in AZ01 were observed as result of existing constraints on AMD v4/v5 and Intel v5 SKUs. What went wrong and why? Azure Resource Manager (ARM) provides a management layer that enables customers to create, update, and delete resources such as VMs. As part of this service management system, ARM depends on several VM management services in the Azure control plane, which in turn rely on an Allocator service deployed per Availability Zone. A recent change to Allocator behavior—designed to improve how the service handles heavy traffic— had been progressively rolled out over the prior two months, expanding to broader regions in early September. This update introduced a new throttling pattern that had performed effectively elsewhere. However, a performance issue affecting a subset of Allocator machines triggered the new throttling logic which, because of the high throughput demand in the East US 2 region, caused VM provisioning requests to retry more aggressively than the control plane could handle. Customer deployments failing in AZ03 were automatically redirected to AZ02, which already had significant load. This redirection, combined with normal traffic, led to overload and throttling in AZ02. Unlike AZ03, which recovered quickly once new VM deployments were halted, AZ02 faced a more complex and persistent challenge. By the time we paused new VM deployments in AZ02, the zone had already accumulated a significant backlog—estimated to be many times larger than normal operational volume. This backlog wasn’t just a result of repeated retries of the same operation. Instead, the platform was attempting a wide variety of retry paths to find successful outcomes for customers. These included retries across multiple internal infrastructure segments within AZ02, triggered by disk and VM co-location logic, and retries with different allocation parameters. Each retry was an attempt to find a viable configuration, not a simple repetition of the same request. This behavior reflects a platform optimized for reliability—one that persistently attempts alternative solutions rather than failing fast. While this approach often helps customers succeed in constrained environments, in this case it led to a large, persistent queue of operations that overwhelmed control plane services in AZ02. The zone’s larger size and higher throughput compared to AZ03 further compounded the issue, making recovery significantly slower. Importantly, the majority of the incident duration was spent in the recovery phase for AZ02. Once new deployments were halted and retries suppressed, the platform still had to drain the backlog of queued operations. This was not a simple in-memory retry loop—it was a persistent queue of work that had accumulated across multiple services and layers. Recovery was further slowed by service-level throttling configurations that, while protective, limited how quickly the backlog could be processed. These throttling values were tuned for normal operations and not optimized for large-scale recovery scenarios. We acknowledge that the extended recovery time in AZ02 was driven by the scale and complexity of the accumulated work, the opportunistic retry behavior of the platform, and the limitations of current throttling and queue-draining mechanisms. Addressing these factors is a key focus of our repair actions—both to prevent excessive backlog accumulation in future incidents, and to accelerate recovery when it does occur. During this incident, our initial communications strategy relied heavily on automated notifications to impacted customers, triggered by monitoring signals for each impacted service. While this system informed many customers quickly, it currently supports only a subset of services and scenarios. As a result, some impacted customers and services were not notified automatically, relying on manual communications provided later during the incident. As the scope of impact widened, we recognized that our communications were not reaching all affected customers. To address this, we sent broad targeted messaging to all customers running VMs in East US 2 – and ultimately published to the (public) Azure Status page, to improve visibility and reduce confusion. Finally, we acknowledge that early updates lacked detail around our investigation and recovery efforts. We should have shared more context sooner, and we have corresponding communication-related repair actions outlined below to improve timeliness and clarity in future incidents. How did we respond? • 09:03 UTC on 10 September 2025 – Customer impact began in AZ03. • 09:13 UTC on 10 September 2025 – Monitoring systems observed a spike in allocation failure rates for Virtual Machines, triggering an alert and prompting teams to investigate. • 09:23 UTC on 10 September 2025 – Initial targeted communications sent to customers via Azure Service Health. Automated messaging gradually expanded as impact increased, and additional service alerts were triggered. • 09:30 UTC on 10 September 2025 – We began manual mitigation efforts in AZ03 by restarting critical service components to restore functionality, rerouting workloads from affected infrastructure, and initiated multiple recovery cycles for the impacted backend service. • 10:02 UTC on 10 September 2025 – Platform services in AZ02 began to throttle requests due to the accumulation of excessive load, following customer deployments being automatically redirected from AZ03, in addition to normal customer load in AZ02. • 11:20 UTC on 10 September 2025 – Platform-initiated retries caused excessive load in AZ02, leading to increased failure rates. • 12:00 UTC on 10 September 2025 – We stopped new VM deployments being allocated in AZ03. • 12:40 UTC on 10 September 2025 – The reduction in load in AZ03 enabled AZ03 to recover. Customer success rates for operations on VMs that were already deployed into AZ03 returned to normal levels. • 13:30 UTC on 10 September 2025 – After verifying that platform-initiated retries had sufficiently drained, we re-enabled new VM deployments to AZ03. We then stopped all new VM deployments in AZ02. • 13:30 UTC on 10 September 2025 – We continued to apply additional mitigations to AZ02 platform services with backlogs of retry requests. These included applying more aggressive throttling, draining existing backlogs, restarting services, and applying additional load management strategies. • 15:05 UTC on 10 September 2025 – Customer success rates for operations on VMs already deployed into AZ02 started increasing. • 16:36 UTC on 10 September 2025 – Broad targeted messaging sent to all customers with Virtual Machines in East US 2. • 16:50 UTC on 10 September 2025 – We started gradually re-enabling traffic in AZ02. • 17:17 UTC on 10 September 2025 – The first update was posted on the public Azure Status page. • 18:50 UTC on 10 September 2025 – After a period of monitoring to validate the health of services, we were confident that the control plane service was restored. • 19:30 UTC on 10 September 2025 – Remaining downstream services reported recovery, following backlog drainage and retry stabilization. Customer impact confirmed mitigated. How are we making incidents like this less likely or less impactful? • We have turned off the recently deployed Allocator behavior, reverting to the previous allocation throttling logic. (Completed) • We have already adjusted our service throttling configuration settings for platform services in the Availability Zones of East US 2, as well as other high-load zones/regions. (Completed) • We are tuning and creating additional throttling levers in control plane services, to help prevent similar increased intra-service call volume load from occurring. (Estimated completion: October 2025) • We are creating a set of internal tools to support the draining of increased backlog for faster service recovery from incident scenarios that require this. (Estimated completion: October 2025) • We are addressing identified gaps in our load testing infrastructure, to help detect these types of issues earlier – before rolling out production changes. (Estimated completion: October 2025) • We are working to improve our communication processes and tools to help eliminate delays notifying impacted customers and supplying actionable information in the future. (Estimated completion: December 2025) • In the longer term, we are improving our end-to-end load management practices, to take a more holistic view of the impacts of any throttling, prioritization, and bursts in demand. (Estimated completion: June 2026) How can customers make incidents like this less impactful? • Note that the 'logical' Availability Zones used by each customer subscription may correspond to different physical Availability Zones - customers can use the Locations API to understand this mapping, to confirm which resources run in the physical AZs referenced above: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/training/modules/design-a-geographically-distributed-application and https://learn.microsoft.com/azure/architecture/patterns/geodes • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/VKY3-PF8

Read the full incident report →

Started Mar 18, 2025, 11:20 PM UTC· resolved Mar 19, 2025, 12:30 AM UTC

1. postmortem Mar 19, 2025, 12:30 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/Z_SZ-NV8 What happened? Between 13:37 and 16:52 UTC on 18 March, and again between 23:20 UTC on 18 March and 00:30 UTC on 19 March, 2025, a combination of a third-party fiber cut, and an internal tooling failure resulted in an impact to a subset of Azure customers with services in our East US region. During the first impact window, immediately after the fiber cut, customers may have experienced intermittent connectivity loss for inter-zone traffic that included AZ03 - to/from other zones, or to/from the public internet. During this time, the traffic loss rate peaked at 0.02% for short periods of time. Traffic within AZ03, as well as traffic to/from/within AZ01 and AZ02, was not impacted. During the second impact window, triggered by the tooling issue, customers may have experienced intermittent connectivity loss – primarily when sending inter-zone traffic that included AZ03. During this time, the traffic loss rate peaked at 0.55% for short periods of time. Traffic entering or leaving the East US region was not impacted, but there was some minimal impact to inter-zone traffic from both of the other Availability Zones, AZ01 and AZ02. Note that the 'logical' availability zones used by each customer subscription may correspond to different physical availability zones. Customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ, see: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings. What went wrong and why? At 13:37 UTC on 18 March 2025, a drilling operation near one of our network paths accidentally struck fiber used by Microsoft, causing an unplanned disruption to datacenter connectivity within AZ03. When fiber cuts impact our networking capacity, our systems are designed to redistribute traffic automatically to other paths. In this instance, we had two concurrent failures happen – before the cut, a datacenter router in AZ03 was down for maintenance and was in the process of being repaired. This combination of multiple concurrent failures impacted a small portion of our diverse capacity within AZ03, leading to the potential for intermittent connectivity issues for some customers. At 13:55 UTC on 18 March 2025, we began mitigating the impact of the fiber cut by load balancing traffic and restoring some of the impacted capacity – customers would have started to see their services recover at this time. Additionally, after the fiber cut and failed isolation, at 14:16 UTC a linecard failed on another router further reducing overall capacity to AZ03. However, as traffic had been re-routed, this further reduction in capacity did not cause any additional customer impact. During the initial mitigation efforts outlined above, our auto-mitigation tool encountered a lock contention problem blocking commands on the impacted devices, failing to isolate all capacity connected to those devices. This failure left some of the impacted capacity un-isolated, and our system did not flag this failed isolation state. Due to some capacity being out of service from the fiber cut, this failed state was not immediately flagged in our systems as the down capacity was not carrying production traffic. At approximately 21:00 UTC, our fiber provider commenced recovery work on the damaged fiber. During the capacity recovery process, at 23:20 UTC, as a result of the failure to isolate all the impacted fiber capacity, as individual fibers were repaired, our recovery systems begin re-sending traffic to the devices connected to the un-isolated capacity, therefore, bringing them back into service without safe levels of capacity. This caused traffic congestion that impacted customers as described above. The traffic congestion within AZ03, due to the tooling failure, triggered an unplanned failure mode on a regional hub router that connects multiple datacenters. By design, our network devices attempt to contain congestive packet loss to capacity that is already impacted. Due to the encountered failure mode, this containment failed on a subset of routers – so congestion spread to neighboring capacity on the same regional hub router, beyond AZ03. This containment failure impacted a small subset of traffic from the regional hub router to AZ1 and AZ2. At this stage, all originally-impacted capacity from the third-party fiber cut was manually isolated from the network – mitigating all customer impact by 00:30 UTC on 19 March. At 01:52 UTC on 19 March the underlying fiber cut was fully recovered. At that time, we completed the test and restoration of all capacity to pre-incident levels by 06:50 UTC on 19 March. How did we respond? • 13:37 UTC on 18 March 2025 – Customer impact began, triggered by a fiber cut causing network congestion which led to customers experiencing packet drops or intermittent connectivity. Our monitoring systems identified the impact immediately, so our on-call engineers engaged to investigate. • 13:45 UTC on 18 March 2025 – Our fiber provider was notified of the fiber cut and prepared for dispatch. • 13:55 UTC on 18 March 2025 – Mitigation efforts began identifying the impacted datacenters and redirecting traffic to healthier routes. • 15:07 UTC on 18 March 2025 – All customers using the East US region were notified about connectivity issues, even if their services were not directly impacted. • 16:52 UTC on 18 March 2025 – Mitigation efforts were successfully completed. All devices affected by the fiber cut were isolated, all customer traffic was using healthy paths and not experiencing congestion. • 23:20 UTC on 18 March 2025 – Customer impact recommenced, due to a tooling failure during the capacity repair process of the initial fiber cut. • 00:30 UTC on 19 March 2025 – This impact was mitigated after isolating the capacity that was incorrectly added by the tooling failure as part of the recovery process. Customers and services would have experienced full mitigation. • 01:52 UTC on 19 March 2025 – The underlying fiber cut was fully restored. We continued to monitor our capacity during the recovery process. • 06:50 UTC on 19 March 2025 – Fiber restoration efforts were completed. The incident was confirmed as mitigated. How are we making incidents like this less likely or less impactful? • We are fixing the tooling failure that caused the devices to be restored to take traffic before they were production ready. (Estimated completion: May 2025) • We are expediting a capacity upgrade within the most impacted datacenter, ahead of a planned technology refresh for all datacenters within this region - to de-risk the impact of multiple concurrent failures. (Estimated completion: July 2025) • In the longer term, we are working to limit the scope of impact further – specifically, to prevent the failure of a device from spreading across availability zones. (Estimated completion: February 2026) How can customers make incidents like this less impactful? • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that predominantly impacted a single region: https://learn.microsoft.com/training/modules/design-a-geographically-distributed-application and https://learn.microsoft.com/azure/architecture/patterns/geodes • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/Z_SZ-NV8

Read the full incident report →

Started Mar 03, 2025, 04:22 PM UTC· resolved Mar 03, 2025, 06:37 PM UTC

1. postmortem Mar 03, 2025, 06:37 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/ZT0X-9B8 What happened? Between 16:22 and 18:37 UTC on 3 March 2025, a subset of customers experienced issues when connecting to Microsoft services through our Toronto edge point-of-presence (PoP) location. This impacted customers connecting to Microsoft 365 and/or Azure services - including but not limited to Virtual Machines, Storage, App Service, SQL Database, Azure Kubernetes Service and Azure Portal access. These resources remained available for customers when connections were completed through other network paths. What went wrong and why? Our global network connects the datacenters across Azure regions with our large network of edge locations. Our edge PoP location in Toronto has paired network devices for redundancy. At 10:58 UTC one of the devices experienced hardware failures – we have since determined this to be due to a memory issue on the line cards for the device – and our automation successfully removed the device from the rotation. While we were performing an initial investigation into the cause of device failure, at 16:22 UTC the redundant device also experienced a hardware failure, due to a similar memory issue on the line cards - the line cards restarted, and did not initialize properly after startup, to forward traffic. As the paired device had already been taken out of rotation, earlier that day, automation was not able to perform normal mitigating actions, so it required human intervention. As such, this redundant device remained in rotation and led to network packets being dropped. During our investigation and evaluation of mitigation options, we considered failing traffic out of this PoP altogether, however this would have been particularly impactful to some customer configurations – for example, those with single-homed ExpressRoute connections – so we opted to keep the location in service. Since the hardware failure was on part of the device, our network engineers isolated the faulty line card from the device, which restored connectivity at 18:37 UTC. After further investigation of the hardware failures, we determined the memory increases were triggered due to an increase in the number of network flows through the device, which reached a scaling limit for these line cards. These occurrences of memory exhaustion coincided with a recent unrelated datacenter network change for specific devices, which began on 27 February 2025. The change increased the routing scale, which has since been paused. We monitor for overall memory on the devices, however the memory on the line cards was not monitored specifically, which led to a delay in our overall response and mitigation processes. As such, upon our initial investigation the overall scope of impact was not evident, which delayed broad customer communications. How did we respond? • 16:22 UTC on 3 March 2025 – Customer impact began, as the redundant device began experiencing the hardware failure, while the paired primary device had already been removed from rotation. • 16:32 UTC on 3 March 2025 – Our automation attempted to bring the paired primary device back into rotation and, due to system load balancing after route convergence, the impact was marginally reduced. • 16:43 UTC on 3 March 2025 – First automated alert raised of potential issues, however the scope of impact was not evident from the alert or initial investigations. • 17:19 UTC on 3 March 2025 – Network engineers investigating the issue took steps to remove congestion from the network peer, which reduced impact. • 17:27 UTC on 3 March 2025 – Additional customer reports received, following delays in submitting support requests via the Azure Portal. • 17:46 UTC on 3 March 2025 – Additional engineering teams were engaged to investigate and mitigate. • 17:59 UTC on 3 March 2025 – We began steps to manually remove the redundant device from rotation. • 18:37 UTC on 3 March 2025 – We isolated the faulty line card from the device. Network traffic was shifted to healthy routes, connectivity was restored, and we monitored the service health to ensure customers had been mitigated. How are we making incidents like this less likely or less impactful? • We have scanned and analyzed for the memory issue across our network. (Completed) • We have updated the configuration on the device, to handle more connections. (Completed) • We have added monitoring for this hardware subcomponent memory utilization, to detect issues like this more quickly. (Completed) • We are improving our notification service for similar issues, to notify impacted customers as soon as our internal monitoring detects the issue. (Estimated completion: March 2025) • In the longer term, we are improving our network design to handle multiple device failures in our edge PoP locations. (Estimated completion for Toronto edge: December 2025) How can customers make incidents like this less impactful? • For ExpressRoute high availability, we recommend operating both primary and secondary connections of ExpressRoute circuits in active-active mode: https://learn.microsoft.com/azure/expressroute/designing-for-high-availability-with-expressroute • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/ZT0X-9B8

Read the full incident report →

Started Feb 25, 2025, 04:42 PM UTC· resolved Feb 26, 2025, 01:15 AM UTC

1. postmortem Feb 26, 2025, 01:15 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/TMS9-J_8 What happened? Between 16:42 UTC on 25 February 2025 and 01:15 UTC on 26 February 2025, a platform issue affected Microsoft Entra ID, which prevented customers from authenticating via Microsoft Entra ID using the following features: • Seamless Single Sign-On (Seamless SSO) https://learn.microsoft.com/entra/identity/hybrid/connect/how-to-connect-sso • Microsoft Entra Connect Sync https://learn.microsoft.com/entra/identity/hybrid/connect/whatis-azure-ad-connect The error was caused by DNS resolution failures when trying to access select Microsoft Entra endpoints related to hybrid identity scenarios. Users were not able to seamlessly sign in to apps via Entra ID and were asked to interactively authenticate. For a subset of clients, authentication was completely blocked. The incident was mitigated for 94% of impacted customers by 18:35 UTC on 25 February 2025, and full mitigation occurred for the remainder of impacted customers at 01:15 UTC on 26 February 2025. What went wrong and why? As part of an ongoing maintenance effort, a change was made which inadvertently removed an intermediate DNS record and associated traffic manager used in the mentioned scenarios. The maintenance effort aimed to clean up interim routing topology, introduced during Microsoft Entra adoption of IPv6 in prior years. The DNS record and Traffic Manager were intermediate infrastructure components in the resolution path for autologon.microsoftazuread-sso.com, which is a domain name used in the Microsoft Entra ID's Seamless SSO feature and in the Entra Connect Sync feature. No other authentication flows were affected. The traffic manager was removed as configuration incorrectly indicated its DNS record was not in use. Similar changes leverage a drift detection system to validate that production DNS zone provisioning reflects the desired configuration. Unfortunately, due to a defect in this safety check, the configuration file was not flagged as out-of-sync. As a result, the DNS record was not correctly identified as in-use, so its removal proceeded. The change was reviewed in an internal change tracking system and correctly identified as high risk. However, this second safety check failed because the potential impact was not accurately identified, leading to the change being mistakenly approved to proceed. The issue was partially mitigated at 18:35 UTC for 94% of affected tenants by restoring configuration of the hostname via a CNAME record. The original DNS record was an A record which does not require clients to recursively resolve a hostname. Certain clients (https://aka.ms/CNAMEs) using Kerberos authentication are sensitive to the hostname being an A record and could not form the Service Principal Name (SPN) correctly, causing 403 Forbidden errors or server timeouts. At 01:05 UTC on 26 February, the configuration was reverted to its exact previous state and the impact was fully mitigated at 01:15 UTC on 26 February. Additionally, initial response to the incident and customer notifications were delayed by misconfiguration of auto-engagement systems and monitors. A feature of our incident management tooling used to request assistance from teams was partially affected. The team was required to leverage backup mechanisms to establish a shared incident bridge and merge investigation workstreams. How did we respond? • 16:42 UTC on 25 February 2025 – Relevant DNS records were inadvertently removed. Gradual onset of impact as the 5-minute DNS Time to Live (TTL) expired. • 17:18 UTC on 25 February 2025 – Investigation started based on internal DNS reachability monitor failures. • 17:40 UTC on 25 February 2025 – We identified and isolated the change that introduced the failure. • 18:35 UTC on 25 February 2025 – Approximately 94% of the customer impact had been mitigated, as the DNS configuration related to this authentication scenario had been partially restored. This change allowed autologon.microsoftazuread.sso.com to resolve again. • 19:16 UTC on 25 February 2025 – First notification posted to Azure Status page banner. • 22:35 UTC on 25 February 2025 – Through customer reports, we identified a subset of affected tenants were still experiencing issues, manifesting as 403 Forbidden errors or time outs. • 01:05 UTC on 26 February 2025 – The configuration for the affected hostname was rolled back to last known good state using an A record. • 01:15 UTC on 26 February 2025 – Traffic fully reverted to regular patterns. How are we making incidents like this less likely or less impactful? • We addressed the misconfiguration affecting our incident response systems by updating our auto-engagement processes to ensure timely notifications and updates are provided throughout the incident lifecycle. (Completed) • We have reviewed all internal monitoring related to outside-in reachability for authentication scenarios, to ensure these trigger escalations at the appropriate severity. (Completed) • We are prioritizing the resolution of the defect in the configuration drift detection system, to de-risk scenarios like this one. (Completed) • We will conduct a technical investigation on the feasibility of introducing non-global (e.g. regionally scoped) endpoints for Seamless SSO authentication scenarios. (Estimated completion: April 2025) • An internal investigation is taking place to audit change management practices and re-verify any changes with a similar risk profile over the last 24 months. A change freeze for DNS and traffic management is in place until this effort is completed. (Estimated completion: March 2025) • Furthermore, we will be upgrading the drift detection system to include additional exhaustive checks of all endpoints used in authentication scenarios. (Estimated completion: April 2025) • Currently, Azure Service Health alerts can only be configured at the Azure subscription level. To enable us to better communicate with our customers, we plan to enable tenant-level Service Health notifications, so that certain communications sent to Tenant Administrators can trigger Azure Service Health alerts. (Estimated completion: October 2025) How can customers make incidents like this less impactful? • Use Managed Identities for Azure Resources in lieu of user accounts employed as service accounts, where possible. Where the use of Managed Identities is not possible, adopt an application-only access pattern, rather than relying on service accounts: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview and https://learn.microsoft.com/entra/identity-platform/app-only-access-primer. • Reevaluate whether Seamless SSO is still required. For customers using Windows 10, Windows Server 2016, or later versions, single sign-on via Windows Primary Refresh Tokens provides a more reliable and secure alternative to Seamless SSO. • As a general best practice, leverage latest authentication libraries such as MSAL, which include safe-by-default implementations of end-to-end authentication flows that account for a range of failure scenarios. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • The impact times above represent the full incident duration, so they are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, ensure that the right people in your organization will be notified about any future service issues by configuring Azure Service Health alerts. These alerts can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts and specifically here to receive Entra ID email notifications: https://learn.microsoft.com/entra/identity/monitoring-health/howto-configure-health-alert-emails How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/TMS9-J_8

Read the full incident report →

Started Jan 08, 2025, 10:31 PM UTC· resolved Jan 11, 2025, 12:44 AM UTC

1. postmortem Jan 11, 2025, 12:44 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/PLP3-1W8 What happened? Between 22:31 UTC on 8 January and 00:44 UTC on 11 January 2025, a networking configuration change in a single availability zone (physical zone AZ01) in East US 2 resulted in customers experiencing connectivity issues, prolonged timeouts, connection drops, and resource allocation failures across multiple Azure services. Customers leveraging Private Link with Network Security Groups (NSG) to communicate with Azure services may have also been impacted. Services that were configured to be zonally redundant and leveraging VNet integration may have experienced impact across multiple zones. The 'logical' availability zones used by each customer subscription may correspond to different physical availability zones. Customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ - https://learn.microsoft.com/en-us/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings . Impacted services included: • Azure Databricks: between 22:55 UTC on 08 January and 15:00 UTC on 10 January, customers may have experienced issues launching clusters and serverless compute resources, leading to failures in dependent jobs or workflows. • Azure OpenAI: between 04:29 UTC on 09 January and 15:00 UTC on 10 January, customers may have experienced stuck Azure OpenAI training jobs, and failures in Finetune model deployments and other jobs. • Azure SQL Managed Instance, Azure Database for PostgreSQL flexible servers, Virtual Machines, Virtual Machine Scale Sets: between 22:31 UTC on 08 January and 22:19 UTC on 10 January, customers may have experienced service management operation failures. • Azure App Service, Azure Logic Apps, Azure Function apps: between 22:31 UTC on 8 January and 20:05 UTC on 10 January, customers may have received intermittent HTTP 500-level response codes, timeouts or high latency when accessing App Service (Web, Mobile, and API Apps), App Service (Linux), or Function deployments hosted in East US 2. • Azure Container Apps: between 22:31 UTC on 08 January and 21:01 UTC on 10 January, customers may have experienced intermittent failures when creating revisions, container app/jobs in a consumption workload profile, and scaling out a Dedicated workload profile in East US 2. • Azure SQL Database: between 01:29 UTC on 09 January and 22:25 UTC on 10 January, customers may have experienced issues accessing services, and errors/timeouts for new database connections. Existing connections remained available to accept new requests, however re-established connections may have failed. • Azure Data Factory and Azure Synapse Analytics: between 22:31 UTC on January 8 and 00:44 UTC on January 11, customers may have encountered errors when attempting to create clusters, job failures, timeouts during Managed VNet activities, and failures in activity runs for the ExecuteDataFlow activity type. Other impacted services included API Management, Azure DevOps, Azure NetApp Files, Azure RedHat OpenShift, and Azure Stream Analytics. What went wrong and why? The Azure PubSub service is a key component of the networking control plane, acting as an intermediary between resource providers and networking agents on Azure hosts. Resource providers, such as the Network Resource Provider, publish customer configurations during Virtual Machine or networking create, update, or delete operations. Networking agents (subscribers) on the hosts retrieve these configurations to program the hosts networking stack. Additionally, the service functions as a cache, ensuring efficient retrieval of configurations during VM reboots or restarts. This capability is essential for deployments, resource allocation, and traffic management in Azure Virtual Network (VNet) environments. Around 22:31 UTC on 08 January, during an operation to enable a feature, a manual recycling of internal PubSub replicas on certain metadata partitions was required. This recycling operation should have been completed in ‘series’ to ensure that data quorum for the partition is maintained. In this case, recycling was executed across the replicas in ‘parallel’, causing the partition to lose quorum. This resulted in the PubSub service to lose the indexing data stored on these partitions. Losing the indexing data on three partitions resulted in ~60% of network configurations not being delivered to agents from the control plane. This impacted all service management operations for resources with index data on the affected partitions, including the network resource provisioning for virtual machines in East US 2, Az01. Networking agents would have failed to receive updates on VM routing information for new and existing VMs that underwent any automated recovery. This manifested in a loss of connectivity affecting VMs that customers use, in addition to the VMs that underpin our PaaS service offerings. Multi-tenant PaaS services that used VNet integration had a dependency on the partitions for the networking provisioning state. After the index data was lost, the dependency on this data that no longer existed caused impact to these PaaS services to expand beyond the impacted zone. How did we respond? The issue was detected by monitoring on the PubSub service at 22:44 UTC, with impacted services raising alerts shortly thereafter. The PubSub team identified the trigger as the three partitions with indexing data loss due to the recycling. At 23:10 UTC, we started evaluating different options for data recovery. Several workstreams were evaluated including: • Rehydrating the networking configurations to the partitions from the network resource provider. • Restoring from backup to recover the networking configurations. • Recovering data from the unaffected partitions. Due to limitations in feasibility for first two options, we started to recover data by rebuilding the mappings form the backend data store. Scanning the data store to extract source mappings is a time intensive process. It involves not only scanning the data, but also making sure that the mappings to the existing resources are still valid. To achieve this, we had to develop validation and mapping transformation scripts. We also throttled the mappings extraction to avoid impacting the backend data store. This scanning process took about 5-6 hours to complete. Once we obtained these mappings, we then had to restore the impacted index partitions. As we initiated the build phase, there were unforeseen challenges that we had to work around. Typically, each build takes 3-4 hours. We spent some time looking for ways to expedite this process, as our plan involved applying multiple builds. Independently on the builds taking longer than expected, we also observed a high number of failure requests due to the outage itself. As a result, our systems were generating a lot of errors which contributed to high CPU usage. This exacerbated the slowness in the restore process. One of the ways that we overcame this this was to deploy patches to block requests to the impacted partitions which mitigated the high CPU usage. Unfortunately, at 3:36 UTC on 10 January 2025, we discovered the first mapping source files that we generated were insufficient as we referenced an incorrect endpoint. Once this was understood, we had to repeat the mapping file generation and restore process. By 19:18 UTC on 10 January, all three partitions were fully recovered. Additional workstreams that were executed while this main workstream was underway included: • An immediate remediation step was to direct the Azure control plane service running on all host machines in this region to PubSub endpoints located in other non-impacted availability zones. This restored the host agent’s ability to look up the correct routing information resulting in mitigating VM to VM connectivity issues. • We initiated a workstream to route all new allocations requests to other availability zones in East US 2. However, as mentioned, the services that were using VNet integration, which had a fixed dependency on the impacted zone, would have continued to see failures. • To remediate impact to customers leveraging Private Link with NSGs to communicate with Azure services, we redirected the Private Link mapping to avoid the dependency on the impacted zone. By 19:18 UTC on 10 January, all three impacted partitions were fully recovered. Following this, we started working with impacted services to validate their health and remediate any data gaps that we discovered. At 00:30 UTC on the 11 January, we initiated a phased approach to gradually enable Az01 to accept new allocation requests. By 00:44 UTC on 11 January, all services confirmed mitigation. We reintroduced Az01 into rotation fully at 04:30 UTC, at which point we declared full mitigation. The impact window for Azure Services varied between service. Below is a timeline of events: • 22:31 UTC on 08 January 2025 – Customer impact begins following the recycling of the PubSub partitions. • 22:44 UTC on 08 January 2025 – The issue is identified as an incorrect networking configuration. • 23:10 UTC on 08 January 2025: We start to evaluate options for the index data recovery to the PubSub service. • 23:43 UTC on 08 January 2025: Targeted customer communications from the earliest reported impacted services started going out via Azure Service Health. • 00:35 UTC on 09 January 2025: We begin scanning the backend data store. • 02:37 UTC on 09 January 2025: Azure Status page updated with the details of a wider networking impact to the region. • 03:05 UTC on 09 January 2025: A workstream to route all new allocation requests to other availability zones in East US 2 is initiated. • 03:45 UTC on 09 January 2025: The workstream for the host agent’s ability to look up the correct routing information was completed. • 04:30 UTC on 09 January 2025: Developing scripts to restore the impacted partitions started. • 05:00 UTC on 09 January 2025: The workstream to route all new allocation requests to other availability zones in East US 2 was completed. • 07:45 UTC on 09 January 2025: Validation and testing of scripts for restoring mapping data to the impacted partitions is completed. • 08:51 UTC on 09 January 2025: Completed scanning backend generated source mapping files. • 09:14 UTC on 09 January 2025: Build process completed for the scripts to restore mappings. • 11:30 UTC on 09 January 2025: We observed high CPU and slow restoration speed. • 11:58 UTC on 09 January 2025: The workstream to mitigate impacted services with Private link and NSG dependencies is initiated. • 13:12 UTC on 09 January 2025: We updated one of the internal PubSub services to stop sending requests which was contributing to the high CPU usage. • 16:10 UTC on 09 January 2025: We completed another update to reduce request volumes to impacted partitions. This helped to improve the restore speed further. • 18:17 UTC on 09 January 2025: The workstream to redirect Private Link mapping to avoid the dependency on the impacted zone is completed. • 19:15 UTC on 09 January 2025: We created another patch to speed up the restore process. The new patch restores the mappings inside the partition rather than calling APIs to add mappings. • 23:40 UTC on 09 January 2025: This patching completed with the new optimization step, and the time for rebuilding one partition was reduced even further. Our overall restore time was reduced by >75%. • 02:10 UTC on 10 January 2025: Restoration of the first two partitions completed. • 03:00 UTC on 10 January 2025: Restoration of the third partition completed. • 03:36 UTC on 10 January 2025: Following the restoration, during our validation steps, we determined that the source mapping files generated were incorrect. As a result, we started generating all new source mapping files. • 05:00 UTC on 10 January 2025: We validated the data of source mappings for first partition. Continued extraction of the rest of the mappings. Meanwhile, we had to clean the previously restored partitions. • 08:30 UTC on 10 January 2025: The correct source mapping file is ready, and we started the partition restore process. • 11:04 UTC on 10 January 2025: The first partition is back online while troubleshooting and mitigating data anomalies. • 19:18 UTC on 10 January 2025 – All three of the impacted partitions are fully recovered and validated – At this stage most customers and services would have seen full mitigation. • 00:30 UTC on 11 January 2025 – We initiated a phased approach to gradually enable the impacted zone for new allocations. • 00:44 UTC on 11 January 2025 - All services confirmed mitigation. How are we making incidents like this less likely or less impactful? • We are working on reducing the recovery time for any data loss scenarios by improving the PubSub data recovery mechanism. This will bring our RTO to under 2 hours (Estimated completion: February 2025). • While PubSub production changes already go through our Safe Deployment Practices (SDP), we have added a step with additional layers of scrutiny. These will include, but are not limited to, verifying rollback plans, ensuring error budgets compliance, and performance validations (Completed). • All manual configuration changes will be moved to an automation pipeline which will require elevated privileges. (Estimated completion: February 2025). • Significant impact to the Pubsub service can occur if certain operations are performed incorrectly. We will block these operations from being executed, except by designated experts (Estimated completion: January 2025). • We are re-examining zonal services to ensure that all downstream dependencies are zone resilient. (Estimated completion: October 2025) • During the incident, we recommended that customers invoke a failover. This was an inappropriate communication. We are making corrections to our processes to ensure that our communications inform customers about the state of the impacted services, zones and regions. This, coupled with any other information that we have about the incident, including ETAs, should be taken into consideration as customers evaluate their own business continuity decisions (Estimated completion: January 2025) How can customers make incidents like this less impactful? • Consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • Consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/PLP3-1W8

Read the full incident report →

Started Dec 26, 2024, 06:40 PM UTC· resolved Dec 27, 2024, 07:30 PM UTC

1. postmortem Dec 27, 2024, 07:30 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/CNF4-N_0 What happened? Between 18:40 UTC on 26 December and 19:30 UTC on 27 December 2024, multiple Azure services were impacted by a power event that occurred in one datacenter, within one Availability Zone (physical zone AZ03), in the South Central US region. Within the impacted datacenter, our automated power systems managed the event as expected, without interruption for two of the three data halls. However, one data hall did not successfully transition to an alternate power supply. This failure led to a loss of compute, network, and storage infrastructure in this data hall. Customer workloads configured for multi-zone resiliency would have seen no impact, or only brief impact, as automated mitigations occurred. Only customer workloads without multi-zone resiliency, and with dependencies on the impacted infrastructure, became degraded or unavailable. Impacted downstream services included: • Azure Alerts Management – between 18:40 UTC 26 December 2024 and 05:00 UTC on 27 December 2024, impacted customers may have experienced high latency in alerts notifications and persistence. • Azure App Service – between 18:40 UTC on 26 December and 12:00 UTC on 27 December, impacted customers may have received intermittent HTTP 500-level response codes, experience timeouts or high latency when accessing App Service (Web, Mobile, and API Apps), App Service (Linux), or Function deployments hosted in the South Central US region. • Azure Application Gateway – between 18:40 UTC on 26 December and 06:58 UTC on 27 December, impacted customers may have experienced data plane disruptions when trying to access your backend applications using Application Gateway hosted in the South Central US region. • Azure Backup – between 20:40 UTC on 26 December and 02:21 UTC on 27 December, impacted customers may have experienced failures in backup operations for Azure File shares in the South Central US region. • Azure Cache for Redis – between 18:45 and 21:35 UTC on 26 December, impacted customers may have lost cache availability and/or been unable to connect to cache resources hosted in the South Central US region. • Azure Cosmos DB – between 18:47 UTC on 26 December and 03:59 UTC on 28 December, impacted customers may have experienced a degradation in service availability and/or request latency. Some requests may have resulted in server errors or timeouts. • Azure Database for PostgreSQL – between 18:48 UTC on 26 December and 13:12 UTC on 27 December, impacted customers may have experienced connectivity failures and timeouts when executing operations, as well as unavailability of resources hosted in the South Central US region. • Azure Database Migration Service – between 19:01 UTC on 26 December and 12:17 UTC on 27 December, impacted customers may have experienced timeout errors when attempting to create a new migration service, or when using existing migration service in the South Central US region. • Azure Event Hubs | Azure Service Bus – Customers with Standard SKU, Premium SKU namespaces or AZ-enabled dedicated Event Hubs clusters experienced an availability drop for approximately five minutes, at the time when the incident started – this issue was mitigated automatically once namespace resources were reallocated to other availability zones. However, a subset of customers using Event Hubs Dedicated non-AZ clusters experienced an availability issue for an extended period of time when trying to access their Event Hubs namespaces in the region. The affected Event Hubs dedicated clusters recovered once the underlying failing VMs in their clusters were brought back online, the last of which were restored by 05:52 UTC on 27 December. • Azure Firewall – between 18:44 UTC on 26 December and 11:30 UTC on 27 December, impacted customers with an Azure Firewall deployed with multi-zone resilience may have seen partial throughput degradation and no availability loss. Customers with an Azure Firewall not utilizing multi-zone resiliency may have had resources dependent on the impacted Availability Zone (physical zone AZ03) which could have resulted in performance degradation or availability impact. Customers attempting control plane operations (for example, making changes to Firewall policies/rules) may have experienced failures during this incident. • Azure Logic Apps – between 18:47 UTC on 26 December and 03:10 UTC on 27 December, impacted customers may have encountered delays in run executions and failing data or control plane calls. • Azure SQL Database – between 20:12 UTC on 26 December and 18:22 UTC on 27 December, impacted customers may have experienced issues accessing services. New connections to databases in the South Central US region may have resulted in an error or timeout. Existing connections remained available to accept new requests, however if those connections were terminated then re-established, they may have failed. • Azure Storage – between 18:45 UTC on 26 December and 08:50 UTC on 27 December, impacted customers may have experienced timeouts and failures when accessing storage resources hosted in the South Central US region. This affected both Standard and Premium tiers of Blobs, Files and Managed Disks. • Azure Synapse Analytics – between 18:53 UTC on 26 December and 13:52 UTC on 27 December, impacted customers may have experienced spark job execution failures, and/or errors when attempting to create clusters, in the South Central US, East US 2, and/or Brazil South regions. • Azure Virtual Machines – between 18:41 UTC on 26 December and 22:26 UTC on 27 December, impacted customers may have experienced connection failures when trying to access some Virtual Machines hosted in the South Central US region. These Virtual Machines may have also restarted unexpectedly. • Azure Virtual Machine Scale Sets – between 19:04 UTC on 26 December and 11:18 UTC on 27 December, impacted customers may have experienced error notifications when performing service management operations - such as create, delete, update, scaling, start, stop - for resources hosted in the South Central US region. • This incident also impacted a subset of Microsoft 365 services – further details are provided in the Microsoft 365 Admin Center, under incident ID MO966473. What went wrong and why? This incident was initially triggered by a utility power loss, itself caused by a localized ground fault – in which a high voltage underground line failed. After a phase to ground short developed in the buried feeder cables, the breaker feeding the datacenter tripped – leading to a loss of utility power, at 18:40 UTC. By design, the power distribution systems transferred power to diesel backup generators, where UPS batteries carry the load during this transition, which was successful for two of the three affected data halls. During the transition to generator power, the third data hall experienced UPS battery faults, which caused the load to drop during transition. In any power-related event, our first priority is to ensure the safety of our staff and infrastructure before any power restoration work can begin. Following our assessment, we were able to safely begin restoration at 20:13 UTC. IT power loads were manually re-energized on backup diesel generator power, by performing a bypass on the failed UPS devices. We began seeing infrastructure services returning by 20:35 UTC, with power fully restored by 20:56 UTC. As power and infrastructure recovered, the next validation steps were to ensure that Azure Networking and Azure Storage services were recovering as expected. By 21:00 UTC, almost all storage and network infrastructure services were confirmed as fully operational. A single storage scale unit remained significantly degraded, due to hardware that required deeper inspection and ultimately, replacement. As storage scale units recovered, 85% of the impacted Virtual Machines (VMs) recovered by 21:40 UTC as their Virtual Hard Disks (VHDs) became available. The next 13% of VMs recovered between 06:00 and 06:30 UTC, as the final storage scale unit became available. Despite all the storage issues being resolved, <2% of VMs impacted by this event remained unhealthy. These issues are detailed below and explain why impacted downstream services with dependencies on these VMs experienced long-tail recoveries. The incident was declared as mitigated at 19:30 UTC on 27 December 2024. Azure Storage: For Zone Redundant Storage (ZRS) accounts, there was no availability impact – as data was served from replicas in other Availability Zones during this incident. The power loss event impacted six Storage scale units. After power restoration, scale units hosting Standard SSD Managed Disks, Premium SSD Managed Disks, Premium Blobs, and Premium Files, fully recovered automatically in around 30 minutes. For most of the HDD-based Standard Storage LRS/GRS scale units, the storage services took approximately one hour to recover. Unfortunately, within one Standard Storage scale unit, multiple network switches were non-functional following the power event, causing a significant portion of the data in that scale unit to be inaccessible because all replicas were unreachable. This caused significant impact to VMs and dependent services that were using Standard HDD managed disks and LRS blob or file storage accounts hosted on this scale unit. Mitigation required replacement networking equipment to be sourced from spares and installed by datacenter technicians. Network engineers then configured and validated these devices, before bringing them online. Additional actions were taken to recover storage nodes under the replaced switches. For the majority of accounts availability was restored by 06:10 UTC on 27th December 2024 (overall availability at 99.5%), with repairs required on a handful of servers to restore 100% availability by 08:50 UTC on 27 December 2024. Azure Compute / Virtual Machines: For customers using VM/compute workloads that leveraged multi-zone resiliency (such as VMSS flex across availability zones), there was no availability impact. For incidents like this, Azure has an automated recovery suite called ‘Defibrillator’ that starts automatically, to recover the VMs and Host machines they are running on, after datacenter power has been restored. It will orchestrate the power on for all affected Host machines, monitor the boot-up and bootstrap sequences, and ensure that the VMs are up and running. When this is running, Azure’s automated steady-state health detection and remediation systems suspend all activities, in order to avoid disrupting the disaster recovery process. At approximately 22:00 UTC on 26 December 2024, some compute scale units were found not tracking at the expected level of recovery. For the final 2% of VMs mentioned above, these experienced an extended recovery – we observed three separate events that contributed to this. • The first scenario was due to initialization without a connection to a network device. Due to the network devices not being fully configured before the Host machines were powered on, a race condition triggered during the Host bootstrap process. This issue is specific to a certain hardware configuration within localized compute scale units, and necessitated the temporary disabling of some validation checks during the bootstrap process. • The second scenario delaying recovery was some machines failing to boot into the Host OS due to a newly discovered bootloader bug impacting a small subset of host hardware with higher levels of offline memory pages. When the hardware reports repeated corrected memory errors to the Host OS, the Host will offline certain memory ranges to prevent repeated use of that memory range. In a small subset of host hardware where a large range of offline memory was accumulated, this new Host OS bug was discovered – resulting in failing to bootstrap the Host OS. This category was mitigated by clearing and/or ignoring this offline memory list and allowing the Host OS to make forward progress where it could, then rebuild its offline memory list once it started to run the full OS. • The third scenario that had prevented compute recovery in some cases was due to control plane devices that are inline to execute the power operations on the Host machines. Datacenter technicians were required to reseat that infrastructure manually. By 10:50 UTC on 27 December, >99.8% of the impacted VMs had recovered, with our team re-enabling Azure’s automated detection and remediation mechanisms. Some targeted remediation efforts were required for a remaining small percentage of VMs, requiring manual intervention to bring these back online. Azure Cosmos DB - For Azure Cosmos DB accounts configured with availability zones, there was no impact, and the account maintained availability for reads and writes. Impact on other Cosmos DB accounts varied depending on the customer database account regional configurations and consistency settings: • Database accounts configured with availability zones were not impacted by the incident, and maintained availability for reads and writes. • Database accounts with multiple read regions and a single write region outside South Central US maintained availability for reads and writes if configured with session or lower consistency. Accounts using strong or bounded staleness consistency may have experienced write throttling to preserve consistency guarantees until the South Central US region was either taken offline or recovered. This behavior is by design. • Active-passive database accounts with multiple read regions and a single write region in South Central US maintained read availability, but write availability was impacted until the South Central US region was taken offline or recovered. • Single-region database accounts in South Central US without Availability Zone configuration were impacted if any partition resided on the affected instances. Azure SQL Database: For Azure SQL Databases configured with zone redundancy, there was no impact. A subset of customers in this region experienced unavailability and slow/stuck control plane operations, such as updating the service level objective, for databases that are not configured as zone redundant. Customers with active geo-replication configuration were asked to consider failing out of the region at approximately 22:31 UTC. Impact duration varied. Most databases recovered after Azure Storage recovered. Some databases took an extended time to recover due to the aforementioned long recovery time of some underlying VMs. Azure Application Gateway: • Application Gateway experienced issues with data path, control plane, and auto-scale operations, leading to service disruptions. Impact on Application Gateways varied depending on customer configuration: • Customers who deployed Application Gateways with zone redundancy may have experienced latency issues and overall degraded performance. • Customers who deployed Application Gateways to a single zone or did not specify zone info during deployment may have experienced data path loss if their deployments had instances in the affected zone. • Gateways with instances deployed in affected zone may have experienced failures or delays in configuration updates. • Gateways with instances deployed in affected zone may have experienced failures or delays in auto scale operations. Azure Firewall: For Azure Firewalls deployed to all Availability Zones of the region, customers would not have experienced any data path impact. However, customers with an Azure Firewall deployed only to the impacted Availability Zone (physical zone AZ03), may have experienced some performance degradation – affecting the ability to scale out. Finally, customers attempting control plane operations (for example, making changes to Firewall policies/rules) may have experienced failures during this incident. Both of these impacts were experienced between 18:40 UTC on 26 December and 07:22 UTC on 27 December 2024. Azure Synapse: Some users of Azure Synapse Analytics faced spark job execution failures in South Central US, Brazil South, and EastUS2. This impacted less than 1% of Synapse calls in those regions. Your logs may include one or more of the following errors that could be a result of this issue: “CLUSTER_CREATION_TIMED_OUT”, “FAILED_CLUSTER_CREATION”, “CLUSTER_FAILED_AFTER_RUNNING”. During this period, Azure Synapse could not provision on-demand compute due to failure to retrieve Management Group's ancestry for RBAC evaluations. The underlying storage for the SCUS instance of the ancestry data was impacted by this incident, which South Central US, Brazil South, and EastUS2 regions depend on. The data is replicated globally and regional failover attempts were made, but did not succeed due to a gateway error. The issue was resolved across all regions once South Central US region was recovered. How did we respond? • 18:40 UTC on 26 December 2024 – Initial power event occurred which led to power loss in the affected data hall. • 18:45 UTC on 26 December 2024 – Technicians from datacenter operations team engaged. • 18:46 UTC on 26 December 2024 – Portal Communications started being sent to impacted subscriptions. • 19:02 UTC on 26 December 2024 – Datacenter incident call began to support triaging and troubleshooting issues. • 19:08 UTC on 26 December 2024 – Azure engineering teams joined a central incident call, to triage and troubleshoot Azure service impact. • 20:13 UTC on 26 December 2024 – Power restoration assessed safe and began. • 20:35 UTC on 26 December 2024 – Compute, Network, and Storage infrastructure began to recover. • 20:54 UTC on 26 December 2024 – Communications published to our public status page. • 20:56 UTC on 26 December 2024 – Power had been restored. Infrastructure recovery continued. • 21:40 UTC on 26 December 2024 – 85% of the VMs impacted by underlying VHD availability recovered. • 06:30 UTC on 27 December 2024 – Additional 13% of VMs impacted by VHD availability recovered. • 08:30 UTC on 27 December 2024 – Ongoing mitigation of additionally impacted services. • 13:00 UTC on 27 December 2024 – Mitigation to most affected services confirmed. • 19:30 UTC on 27 December 2024 – Incident mitigation confirmed and declared. How are we making incidents like this less likely or less impactful? • Datacenter response to return to utility power after ensuring battery health for UPS transition from generator (Completed). • We are reviewing the nature of the UPS battery failures in line with our global battery standards and maintenance procedures, to identify improvements to de-risk this class of issue across the fleet. (Estimated completion: February 2025) • Repairs to the offline failed utility line are in progress. (Estimated completion: February 2025) • The mitigation to bypass various checks during the bootstrap process have been applied to all impacted machines and are being evaluated and executed for other hardware configurations where needed. (Estimated completion: March 2025) How can customers make incidents like this less impactful? • Consider using Availability Zones (AZs) to run your services across physically separate locations within an Azure region. To help services be more resilient to datacenter-level failures like this one, each AZ provides independent power, networking, and cooling. Many Azure services support zonal, zone-redundant, and/or always-available configurations: https://docs.microsoft.com/azure/availability-zones/az-overview • Note that the 'logical' Availability Zones used by each customer subscription may correspond to different physical Availability Zones - customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ - https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations • Customers using Azure Virtual Machines should consider reviewing our reliability guidance, including Availability Zone support and disaster recovery. See: https://learn.microsoft.com/azure/reliability/reliability-virtual-machines#availability-zone-support • Customers using Azure Cosmos DB should consider reviewing our high availability and reliability guidance for adopting Availability Zone architecture: https://learn.microsoft.com/azure/reliability/reliability-cosmos-db-nosql • Customers using Azure SQL DB should consider reviewing our high availability and resiliency guidance surrounding relevant redundancy: https://learn.microsoft.com/azure/azure-sql/database/high-availability-sla-local-zone-redundancy?view=azuresql&tabs=azure-powershell • Customers currently using Application Gateway v1 should consider migrating to Application Gateway v2, see: https://learn.microsoft.com/azure/application-gateway/migrate-v1-v2 • Customers using Application Gateway v2 should consider ensuring that two or more Azure availability zones are selected during provisioning, for better resiliency. See: https://learn.microsoft.com/azure/application-gateway/overview-v2 • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/CNF4-N_0

Read the full incident report →

Started Nov 13, 2024, 12:50 AM UTC· resolved Nov 13, 2024, 12:30 PM UTC

1. postmortem Nov 13, 2024, 12:30 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/PSM0-BQ8 What happened? Between 00:50 UTC and 12:30 UTC on 13 November 2024, a subset of Azure Blob Storage and Azure Data Lake Storage accounts experienced connectivity errors. The issue manifested as loss of access to Blob and Data Lake storage endpoints of the affected storage accounts, and subsequent unavailability of downstream services that depended on these storage accounts. Since many of the impacted storage accounts were used by other Azure services and major software vendor solutions, the customer impact was widespread. Although unavailable to access, the data stored in these storage accounts was not impacted during this incident. Impacted downstream services included: • Azure Storage: Impacted customers may have experienced name (DNS) resolution failures when interacting with impacted storage accounts in Australia East, Australia Southeast, Brazil South, Brazil Southeast, Canada Central, Canada East, Central India, Central US, East Asia, East US, East US 2, East US 2 EUAP, France Central, Germany West Central, Japan East, Japan West, Korea Central, North Central US, North Europe, Norway East, South Africa North, South Central US, South India, Southeast Asia, Sweden Central, Switzerland North, UAE North, UK South, UK West, West Central US, West Europe, West US, West US 2, West US 3. • Azure Container Registry: Impacted customers using the East US region may have experienced intermittent 5xx errors while trying to pull images from the registry. • Azure Databricks: Impacted customers may have experienced failures with launching clusters and serverless compute resources in Australia East, Canada Central, Canada East, Central US, East US, East US 2, Japan East, South Central US, UAE North, West US, and/or West US 2. • Azure Log Analytics: Impacted customers using the West Europe, Southeast Asia, and/or Korea Central regions may have experienced delays and/or stale data when viewing Microsoft Graph activity logs. What went wrong and why? Azure Traffic Manager manages and routes blob and data lake storage API requests. The incident was caused by an unintentional deletion of Traffic Manager profiles for the impacted storage accounts. These Traffic Manager profiles were originally part of an Azure subscription pool which belonged to the Azure Storage service. This original service was bifurcated to become two separate services, with one service that would eventually be deprecated. Ownership of the subscriptions containing the Traffic Manager profiles for storage accounts should have been assigned to the service that was continuing operation, which was missed. As such, the decommissioning process inadvertently deleted the Traffic Manager profiles under the subscription, leading to loss of access to the affected storage accounts. To learn more about Azure Traffic Manager profiles, see: https://learn.microsoft.com/azure/traffic-manager/traffic-manager-manage-profiles. How did we respond? After receiving customer reports of issues, our team immediately engaged to investigate. Once we understood what had triggered the problem, our team initiated started to restore Traffic Manager profiles of the affected storage accounts. The recovery took an extended period of time since it required care reconstructing the Traffic Manager profiles while avoiding further customer impact. We started multiple workstreams in parallel, to drive both manual recovery and create automated steps to speed up recovery. Recovery was carried out in phases, with the majority of affected accounts restored by 06:24 UTC - and the last set of storage accounts recovered and fully operational by 12:30 UTC. Timeline of key events: • 13 November 2024 @ 00:50 UTC – First customer impact, triggered by the deletion of a Traffic Manager profile. • 13 November 2024 @ 01:24 UTC – First customer report of issues, on-call engineering team began to investigate. • 13 November 2024 @ 01:40 UTC – We identified that the issues were triggered by the deletion of Traffic Manager profiles. • 13 November 2024 @ 02:16 UTC – Impacted Traffic Manager profiles identified, and recovery planning started. • 13 November 2024 @ 02:25 UTC – Recovery workstreams started. • 13 November 2024 @ 03:51 UTC – First batch of storage accounts recovered and validated. • 13 November 2024 @ 06:00 UTC – Automation to perform regional recovery in place. • 13 November 2024 @ 06:24 UTC – Majority of recovery completed; most impacted accounts were accessible by this time. • 13 November 2024 @ 12:30 UTC – Recovery and validation 100% complete, incident mitigated. How are we making incidents like this less likely or less impactful? • We completed an audit of all the production service artifacts used by the Azure Storage resource provider. (Completed) • We created a new highly restrictive deployment approval process, as an additional measure to prevent unintended mutations like deletions. (Completed) • We are improving the process used to clean-up production service artifacts with built-in safety to prevent impact. (Estimated completion: December 2024) • We are enhancing our monitoring of outside-in storage traffic by making it more sensitive to smaller impacts and to validate connectivity and reachability for all endpoints of storage accounts. (Some services completed, all service will complete in December 2024) • We are expanding and completing the process of securing platform resources with resource locks, as an additional safety automation to prevent deletes. (Estimated completion: January 2025) • We will accelerate recovery times by refining restore points and optimizing the recovery process for production service artifacts. (Estimated completion: January 2025) • We will reduce the blast radius with service architectural improvements to improve the resiliency against issues related to traffic manager, and other upstream dependencies’, unavailability. (Estimated completion: March 2025) How can customers make incidents like this less impactful? • Consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/PSM0-BQ8

Read the full incident report →