Microsoft Azure Outage History

Started Oct 08, 2026, 09:58 AM UTC· resolved Oct 08, 2026, 11:28 AM UTC

1. monitoring Oct 08, 2026, 09:58 AM UTC

   Azure Data Factory issues in UK SouthWe are aware of an issue affecting Azure Data Factory in UK South, where a subset of customers may experience failures, delays, or timeouts when running pipelines and activities, managing integration runtimes, or performing authoring operations. Our engineers are actively investigating the cause and impact.Workaround: Customers are advised to cancel and retry affected pipeline or activity runs that fail, time out, or remain in progress longer than expected.Further updates will be provided as more information becomes available.

2. monitoring Oct 08, 2026, 10:03 AM UTC

   We are aware of an issue affecting Azure Data Factory in UK South, where a subset of customers may experience failures, delays, or timeouts when running pipelines and activities, managing integration runtimes, or performing authoring operations. Our engineers are actively investigating the cause and impact . Workaround: Customers are advised to cancel and retry affected pipeline or activity runs that fail, time out, or remain in progress longer than expected. Further updates will be provided as more information becomes available.

3. monitoring Oct 08, 2026, 10:11 AM UTC

   We are aware of an issue affecting Azure Data Factory in UK South, where a subset of customers may experience failures, delays, or timeouts with pipeline and activity runs, integration runtime management, or authoring operations. Our engineers are actively working to determine the cause and impact.As a workaround, customers are advised to cancel and retry affected runs that fail, time out, or remain in progress longer than expected

4. monitoring Oct 08, 2026, 10:12 AM UTC

   We are aware of an issue affecting Azure Data Factory in UK South, where a subset of customers may experience failures, delays, or timeouts with pipeline and activity runs, integration runtime management, or authoring operations. Our engineers are actively working to determine the cause and impact. As a workaround, customers are advised to cancel and retry affected runs that fail, time out, or remain in progress longer than expected.

Read the full incident report →

Started Sep 30, 2026, 08:30 PM UTC· resolved Oct 01, 2026, 02:15 AM UTC

1. monitoring Sep 30, 2026, 08:30 PM UTC

   We are investigating an alert for Azure ExpressRoute Gateway in multiple Azure regions. We will provide more information as it becomes available.

2. monitoring Sep 30, 2026, 08:30 PM UTC

   Starting at 20:30 UTC on 30 September 2026, customers using ExpressRoute and/or Azure VPN Gateway may experience degraded or interrupted connectivity. We are also seeing impact to components responsible for managing network gateways, which may affect some management operations. Our investigation has identified a correlation between the onset of impact and infrastructure ‘operating system’ servicing activity. During this activity, some network gateway instances became unhealthy or temporarily unavailable. We have observed different recovery behavior across the affected services. Many impacted ExpressRoute gateways are recovering as the servicing operation completes. However, some supporting management components have not recovered automatically, and we are actively working to restore those components. Our current working hypothesis is that infrastructure operating system servicing is triggering an unexpected condition in some network service instances. We have not yet confirmed the underlying failure mechanism and are continuing to investigate why affected instances became unhealthy and why recovery behavior differs between services. We have paused further operating system servicing associated with this activity to prevent new instances from entering the affected servicing workflow. In parallel, our engineering teams are restoring network management components that have not recovered automatically as well as evaluating additional recovery actions for resources that remain unhealthy. We are continuing to monitor recovery across the affected services and will provide additional information within the next 30 minutes or as events warrant.

3. monitoring Sep 30, 2026, 08:30 PM UTC

   Starting at 20:30 UTC on 30 September 2026, customers using ExpressRoute and/or VPN Gateway may experience degraded or interrupted network connectivity. Customers may also experience failures or delays when performing some network management operations. Current status: We have paused the infrastructure servicing activity associated with the onset of this event, preventing additional resources from entering the affected servicing workflow. We are seeing continued recovery across impacted ExpressRoute gateways and are monitoring this recovery to confirm that it remains durable. For VPN Gateway, some affected gateways have experienced reduced redundancy rather than a complete loss of connectivity. Recovery continues to be monitored. Some supporting network management components have not recovered automatically and are contributing to management operation failures in a subset of affected regions. Engineering teams are actively restoring these components using alternate healthy instances where available. Our investigation continues to show a correlation between this event and infrastructure operating system servicing activity. However, the underlying failure mechanism has not yet been confirmed. We are investigating why some network service instances became unhealthy and why recovery behavior has differed between services. We will provide additional information within the next 30 minutes, or sooner if events warrant.

4. postmortem Oct 01, 2026, 02:15 AM UTC

   Join either of our upcoming 'Azure Incident Retrospective' livestreams discussing this incident (to hear from our engineering leaders, and to get any questions answered by our experts) or watch a recording of the livestream (available the following week, on YouTube): • Option #1 - 18:30 to 19:00 UTC on 22 October 2026 - https://aka.ms/air/7Q30-010/1 • Option #2 - 06:30 to 07:00 UTC on 23 October 2026 - https://aka.ms/air/7Q30-010/2 What happened? Between 20:30 UTC on 30 September 2026 and 02:15 UTC on 1 October 2026, customers using several Azure networking services – including Azure ExpressRoute Gateway, Azure Application Gateway, Azure Firewall, Azure Bastion, Azure Virtual WAN, and Azure VPN Gateway – experienced intermittent failures when performing service management operations. In addition, a subset of upgraded Azure ExpressRoute gateways could not retrieve their required configuration from the control plane service, resulting in temporary connectivity loss. Customers using Azure VMware Solution may also have temporarily lost network connectivity to their software-defined datacenter environments. The issue affected multiple regions including Australia Central 2, Australia Southeast, Austria East, Belgium Central, Canada East, Central US, Chile Central, Denmark East, East Asia, France Central, Germany North, Germany West Central, India South Central, Indonesia Central, Japan West, Jio India Central, Korea Central, Malaysia West, North Europe, Norway West, South Africa North, South Central US, South India, Southeast Asia, Sweden South, Switzerland North, Taiwan North, UAE North, UK South, West Europe, West US, West US 2, and West US 3. The issue began when a platform-triggered gateway update resumed across multiple regions, increasing request volume to the regional gateway control plane service, thereby affecting its ability to process management operations reliably. Impact times varied region by region, as the issue was triggered by a rolling update moving through the regions. We paused this update while applying a configuration change in phases to restore the control plane service. The last wave of customer impact was mitigated at 02:15 UTC on 1 October 2026. What went wrong and why? Azure regularly applies operating system and security updates to networking gateway services. These updates are designed to roll out gradually across deployment stages and redundant gateway instances, allowing service health to be evaluated before updates proceed. Before this incident, a temporary deployment restriction had paused operating system updates for a group of networking gateways. When this restriction expired, a large number of gateway updates resumed within a short period, generating a surge in gateway restarts and configuration-retrieval requests. At the same time, a previously deployed change had increased the processing time required for configuration operations handled by a shared regional networking control plane service. The combination of increased request volume and longer processing times overwhelmed the control plane, affecting its ability to process requests reliably. As a result, management operations for several Azure networking services failed intermittently. Existing request-throttling and load-protection mechanisms did not sufficiently protect critical processing under these conditions. For Azure ExpressRoute Gateway, an additional issue prolonged connectivity impact. Gateway instances restarting during the update process were marked healthy before their configuration and routing state had been fully restored. This allowed updates to proceed to another redundant instance before the first instance was ready. Consequently, some gateways temporarily lost connectivity to connected on-premises networks because neither instance could retrieve the required routing configuration. After approximately 30 minutes of unsuccessful configuration-retrieval attempts, the affected gateways automatically applied their locally stored last known good (LKG) configurations and restored connectivity. Automated monitoring detected ExpressRoute connectivity degradation at 20:43 UTC on 30 September 2026, followed by broader control plane degradation at 20:58 UTC. We pursued several recovery approaches in parallel, including pausing further gateway updates at 21:45 UTC, shifting control plane traffic, and attempting to add resources. Although these actions provided temporary relief, they did not fully resolve the issue. Further investigation identified the combination of elevated request volume and increased processing time per operation as the primary contributing factors. Once this was understood, we began rolling back the change that had increased control plane processing time, starting at 01:15 UTC on 1 October 2026. The rollback was applied across the affected regions, and customer impact ended at 02:15 UTC. How did we respond? • 20:30 UTC on 30 September 2026 – Customer impact began, with intermittent failures affecting management operations across multiple Azure networking services. • 20:43 UTC on 30 September 2026 – ExpressRoute monitoring detected connectivity degradation. • 20:58 UTC on 30 September 2026 – Control plane monitoring detected health failures across additional regions. • 21:12 UTC on 30 September 2026 – Multiple teams were engaged to investigate the issue and coordinate response efforts. • 21:45 UTC on 30 September 2026 – Further gateway operating system updates were paused to prevent additional update activity. • 22:09 UTC on 30 September 2026 – Traffic shifting actions provided temporary relief, but elevated resource utilization returned. Further traffic shifts were stopped. • 22:15 UTC on 30 September 2026 – Attempts to add control plane resources were investigated, and constraints involving dependent platform services were addressed. • 22:36 UTC on 30 September 2026 – Affected ExpressRoute gateways automatically restored connectivity using their last known good configurations. • 23:55 UTC on 30 September 2026 – We identified increased processing time for gateway configuration operations were associated with a previously deployed performance change, so we determined that a rollback would be a relevant mitigation. • 01:15 UTC on 1 October 2026 – The rollback began in an initial region, where the results were validated before proceeding. • 02:15 UTC on 1 October 2026 – The rollback was completed across the affected regions, and customer impact ended. How are we making incidents like this less likely or less impactful? • We have rolled back the change that introduced the performance regression, to de-risk this failure mode immediately. (Completed) • We are improving how paused operating system updates resume, to ensure they follow staged rollout practices rather than resuming across large numbers of gateways at once. (Estimated completion: October 2026) • We are strengthening ExpressRoute gateway health checks, to ensure that redundant instances are updated only after the initial instances have fully restored its configuration and routing state. (Estimated completion: March 2027) • We are addressing platform issues that prevented the control plane service from scaling out successfully during the incident. (Estimated completion: March 2027) • In the longer term, we are improving throttling and load-shedding mechanisms in the control plane service to help protect critical request processing during unexpected increases in demand. (Estimated completion: June 2027) How can customers make Incidents like this less impactful? • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the overall incident duration and are not specific to any individual customer. Actual impact varied by service, region, customer, and resource. For guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization are notified about any future service issues by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/7Q30-010

5. postmortem Oct 01, 2026, 02:15 AM UTC

   Incorrect Region: correction: Impacted region(s) within a previous communication has been corrected. What happened? Between 20:30 UTC on 30 September 2026 and 02:15 UTC on 01 October 2026, a subset of customers using gateway services in multiple regions experienced degraded or interrupted network connectivity. Impacted customers may have observed gateways failing to load in the Azure Portal, along with failures or delays in network management operations across the following impacted services. • Azure ExpressRoute Gateway • Azure Firewall • Azure Application Gateway and Web Application Firewall • Azure VPN Gateway • Azure VMware Solution What do we know so far? Our investigation identified that a recent change to a regional gateway management service triggered a higher-than-expected load when an unrelated operating system servicing maintenance proceeded gradually through multiple regions. Normally, this gateway management service would auto scale as needed. Demand on dependent services increased due to the increased workload, preventing these regional services from scaling as expected. The operating system servicing maintenance was paused as a precaution. We reverted the contributing regional gateway manager change, which reduced gateway manager load and allowed affected services to recover. We are continuing to investigate the scaling behavior and the safeguards needed to help prevent recurrence. How did we respond? • 20:30 UTC on 30 September 2026 – Customer impact began. • 21:29 UTC on 30 September 2026 – We began investigating connectivity issues affecting Azure ExpressRoute Gateway in UK South. • 22:27 UTC on 30 September 2026 – We identified that the issue was affecting multiple Azure regions and expanded our investigation. • 23:05 UTC on 30 September 2026 – We identified a correlation with operating system servicing activity and paused further servicing while we worked to restore affected services. • 01:36 UTC on 01 October 2026 – Recovery progressed across most affected regions, while configuration changes were applied to remaining impacted regions, including France Central, North Europe, Southeast Asia, UK South, and UK West. • 02:15 UTC on 01 October 2026 – We confirmed that the issue was mitigated. What happens next? • Our team will be completing an internal retrospective to understand the incident in more detail. Once that is completed, generally within 14 days, we will publish a Post Incident Review (PIR) to all impacted customers. • To get notified when that happens, and/or to stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts. • For more information on Post Incident Reviews, refer to https://aka.ms/AzurePIRs. • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability may vary between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, for broader guidance on preparing for cloud incidents, refer to https://aka.ms/incidentreadiness.

Read the full incident report →

Started Sep 29, 2026, 10:00 AM UTC· resolved Sep 29, 2026, 03:58 PM UTC

1. monitoring Sep 29, 2026, 10:00 AM UTC

   We are investigating an issue affecting Azure OpenAI and Foundry agent services in the Sweden Central region. Impacted customers may experience intermittent errors when making service requests. We are investigating the underlying issue and exploring mitigation options to restore normal service performance. Our analysis remains ongoing, and we are closely monitoring service health while continuing our investigation. We will provide additional information as it becomes available.

2. monitoring Sep 29, 2026, 10:00 AM UTC

   We are investigating an issue affecting Azure OpenAI Service, Azure AI Foundry Agent Service, Azure AI Foundry Models, and Azure AI Cognitive Services in the Sweden Central region. Impacted customers may experience intermittent request failures, increased latency and HTTP 5XX error codes when making service requests. We are investigating the underlying issue and exploring mitigation options to restore normal service performance. Our analysis remains ongoing, and we are closely monitoring service health while continuing our investigation. We will provide additional information as it becomes available.

3. monitoring Sep 29, 2026, 10:00 AM UTC

   Impact Statement: Starting at 10:00 UTC on 29 September 2026, a subset of customers using OpenAI Service, Foundry Agent Service, Foundry Models, and Cognitive Services, in Sweden Central may experience intermittent request failures, increased latency, and HTTP 5XX error codes when submitting requests to affected models and APIs hosted in this region. Current Status: Currently we are monitoring improvements in the overall experience. Initial findings indicate that a backend service which is relied on to retrieve metadata information when managing requests for this region, was experiencing timeouts when reading from dependent database and caching layers. As a result, requests that depend on these processes were delayed or failed intermittently. Queued requests are being processed. We are continuing to investigate the health of supporting platform infrastructure or system processes that may impact these requests. The next update will be provided within 60 minutes, or as events warrant.

4. postmortem Sep 29, 2026, 03:58 PM UTC

   Join either of our upcoming 'Azure Incident Retrospective' livestreams discussing this incident (to hear from our engineering leaders, and to get any questions answered by our experts) or watch a recording of the livestream (available the following week, on YouTube): • Option #1 - 17:30 to 18:00 UTC on 22 October 2026 - https://aka.ms/air/7QL5-Z50/1 • Option #2 - 05:30 to 06:00 UTC on 23 October 2026 - https://aka.ms/air/7QL5-Z50/2 What happened? Between 10:03 UTC and 15:58 UTC on 29 September 2026, a platform issue resulted in impact to Azure OpenAI Service, Foundry Agent Service, Foundry Models, and Azure AI services in the Sweden Central region. Impacted customers experienced intermittent request failures, increased latency, and/or HTTP 500 and 503 errors when submitting requests to affected models and data plane APIs hosted in this region. Note that the Azure OpenAI Service processes customer requests through model engines deployed across multiple Azure regions and supported by traffic routing systems. Depending on the selected deployment model (Global, Data Zone, or Regional), note that requests may be routed across multiple Azure regions within defined geographic boundaries to support availability and resilience, while customer data remains stored at rest in the selected Azure region. Learn more at: https://azure.microsoft.com/blog/enterprise-trust-in-azure-openai-service-strengthened-with-data-zones/ What went wrong and why? When requests are made to Azure OpenAI Service and related Azure AI services in a region, the service relies on a regional backend service to retrieve resource metadata and configuration information for the account making the request. This backend service reads from a caching layer and a database layer. Other internal services that call this backend service maintain their own caches of this information, to reduce load on the backend service. At approximately 09:38 UTC on 29 September 2026, a planned deployment of one of these internal services began in Sweden Central. As part of this deployment, the internal service's caches were cleared and - because the deployment coincided with peak traffic hours in the region - this produced a large number of cache misses, where requests were made for information no longer stored in the cache. Two other defects in how that service read from its caches amplified this effect, causing most lookups to bypass the cache and fall back to calling the backend service directly. By approximately 10:00 UTC, traffic to the backend service had increased to approximately 20 times its normal level. This surge drove a sharp increase in requests from the backend service to its caching layer, which reached full utilization, thereby increasing latency. This higher latency caused connections to be held open longer, while client retries added further requests. As a result, the network connection capacity on the infrastructure node instances was exhausted, so new outbound connections to both the caching layer and the database layer could not be established. While this traffic surge was a trigger, the extended duration of the incident was primarily caused by how the backend service determined whether its instances were healthy. Automated health checks for each instance depended on successfully connecting to the database layer. When instances could not establish new outbound connections, they failed this health check and were automatically restarted by design. As more instances entered this restart loop, fewer healthy instances remained to serve requests, and retries from dependent services further increased the load on the remaining instances. Multiple factors also extended the duration of the incident. Existing network metrics and diagnostic metrics did not effectively indicate the connection exhaustion issue in this scenario. It took an extended amount of time to correlate the failures with what was causing this specific connection exhaustion scenario. As a result, different mitigation attempts - including scaling the number of service instances, and increasing compute and memory resources - could not resolve the incident, until the health check issue was identified and mitigated. How did we respond? • 09:38 UTC on 29 September 2026 – A deployment of an internal service in Sweden Central began, which cleared its caches. • 10:00 UTC on 29 September 2026 – Traffic to the backend service began increasing sharply. • 10:03 UTC on 29 September 2026 – Customer impact began because of service instance instability. • 10:17 UTC on 29 September 2026 – Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region, and alerted our engineering teams. • 10:28 UTC on 29 September 2026 – Engineers were engaged and investigating the issue. • 11:21 UTC on 29 September 2026 – Additional engineering teams were engaged, ongoing deployments were reviewed for the Azure OpenAI Service in Sweden Central to determine whether any recent change correlated with the issue. • 11:30 UTC on 29 September 2026 – We identified that infrastructure node service instances were experiencing timeouts when attempting to connect to the backend service's caching and database layers. • 13:55 UTC on 29 September 2026 – We confirmed that the database layer was healthy. We made attempts to mitigate through different scaling adjustments of service instances to lower the number of outbound connections, which did not result in significant improvements. • 15:00 UTC on 29 September 2026 – We further increased compute resources for the backend service and increased its resource limits. In parallel, we investigated infrastructure health for affected service instances and restarted service instances. • 15:40 UTC on 29 September 2026 – We reconfigured the automated health check to no longer depend on remote connectivity, which stopped the restart loop. Service availability began to recover. • 15:58 UTC on 29 September 2026 – Service availability had recovered, and customer impact was mitigated. • 16:30 UTC on 29 September 2026 – We identified the source of the increased traffic and correlated it with the internal service deployment, while monitoring confirmed that services were stable. How are we making incidents like this less likely or less impactful? • We have reconfigured the automated health check for the backend service so that it no longer depends on connectivity to its database layer, preventing instances from being restarted unnecessarily during dependency issues. (Completed) • We have added monitoring and alerting on the load and latency of the caching layer used by the backend service. (Completed) • We have updated our troubleshooting and procedural guidance for such connection exhaustion scenarios, to assist on-call engineers in mitigating similar scenarios. (Completed) • We are adding additional throttling and rate limiting for the backend service, to protect it from sudden increases in traffic. (Estimated completion: October 2026) • We are increasing the resource capacity of the caching layer used by the backend service across regions. (Estimated completion: October 2026) • We are improving diagnostic metrics and automated detection, to help identify such issues more quickly. (Estimated completion: October 2026) • In the longer term, we are partitioning the caching layer used by the backend service to increase its throughput. (Estimated completion: December 2026) How can customers make incidents like this less impactful? • Consider reviewing our guidance and best practices related to Business Continuity and Disaster Recovery (BCDR) scenarios for Microsoft Foundry: https://learn.microsoft.com/azure/foundry/how-to/high-availability-resiliency • Consider reviewing and implementing our best practices surrounding retry patterns, especially with exponential backoff, to improve workload resiliency during intermittent issues: https://learn.microsoft.com/azure/architecture/patterns/retry • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/7QL5-Z50

Read the full incident report →

Started Sep 03, 2026, 10:03 AM UTC· resolved Sep 03, 2026, 10:36 AM UTC

1. monitoring Sep 03, 2026, 10:03 AM UTC

   We are investigating an issue affecting a subset of infrastructure within the East US region. Customers may experience service degradation for dependent Azure services. Our teams are actively investigating the cause and working to restore normal operations. More information will be provided as it becomes available.

Read the full incident report →

Started Aug 28, 2026, 06:19 AM UTC· resolved Aug 28, 2026, 07:01 AM UTC

1. monitoring Aug 28, 2026, 06:19 AM UTC

   We are investigating an issue impacting multiple Azure services in the West US regions. Engineering teams are actively assessing the scope and working to mitigate customer impact. Further updates will be shared as they become available

Read the full incident report →

Started Aug 25, 2026, 04:36 AM UTC· resolved Aug 25, 2026, 06:11 AM UTC

1. monitoring Aug 25, 2026, 04:03 AM UTC

   We are investigating an issue affecting a subset of infrastructure within the Southeast Asia region. Customers may experience connectivity issues or service degradation for dependent Azure services. Engineers are actively investigating the cause and working to restore normal operations. More information will be provided as it becomes available.

2. monitoring Aug 25, 2026, 04:36 AM UTC

   We have observed recovery for a portion of the infrastructure affected by this issue within the Southeast Asia region. While many impacted services and resources have recovered, a subset of customers may continue to experience connectivity issues or service degradation for dependent Azure services. Engineers are actively working to restore the remaining impacted infrastructure and validate full service recovery.More information will be provided as it becomes available.

3. resolved Aug 25, 2026, 06:09 AM UTC

   We have mitigated a power-related infrastructure issue that affected a subset of Azure services in Southeast Asia. Most services have recovered and are operating normally. A remaining issue affecting a single storage stamp is being tracked separately, and impacted customers are being communicated with through a dedicated tracking process.

Read the full incident report →

Started Aug 14, 2026, 05:28 AM UTC· resolved Aug 14, 2026, 06:22 AM UTC

1. monitoring Aug 14, 2026, 05:28 AM UTC

   We are aware of an issue affecting access to the Azure portal in the Australia regions. Our engineers are actively investigating the cause and impact. Further updates will be provided as more information becomes available.

Read the full incident report →

Started Jul 23, 2026, 04:29 PM UTC· resolved Jul 23, 2026, 08:24 PM UTC

1. monitoring Jul 23, 2026, 04:29 PM UTC

   We are investigating a networking issue affecting connectivity to Azure services in the West US region. Impacted customers may experience intermittent connectivity failures, increased latency, or difficulty accessing Azure services. Customers with traffic traversing the West US region may also experience downstream impact. We are exploring mitigation options on affected network devices. Early indications suggest the issue is related to network traffic flow through the impacted infrastructure; however our analysis remains ongoing. We are closely monitoring traffic while continuing our investigation. We will provide additional information as it becomes available.

Read the full incident report →

Started Jul 23, 2026, 02:44 PM UTC· resolved Jul 23, 2026, 04:34 PM UTC

1. monitoring Jul 23, 2026, 03:50 PM UTC

   We are investigating an issue affecting connectivity to Azure services for a subset of customers in the West US region. Impacted customers may experience intermittent issues when accessing Azure services.We will provide more information as it becomes available.

2. postmortem Jul 23, 2026, 07:41 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/ZJV6-SGG What happened? Between 14:44 and 19:41 UTC on 23 July 2026, a subset of customers experienced connectivity failures, increased latency, and/or difficulty accessing Azure services hosted within the West US region. Impact was limited to network traffic entering or exiting the region, traffic that remained entirely within the region was not affected. Impacted services included: App Service, Application Insights, Azure AD B2C, Azure AI Bot Service, Azure AI Search, Azure API Management, Azure Application Gateway, Azure Bastion, Azure Cosmos DB, Azure Data Explorer, Azure Database for PostgreSQL, Azure Databricks, Azure ExpressRoute, Azure Firewall, Azure Kubernetes Service (AKS), Azure Monitor, Azure Speech in Foundry Tools, Azure Virtual Desktop, Azure Virtual WAN, Azure VMware Solution, Azure VPN Gateway, as well as Microsoft Sentinel and Power BI Embedded. Downstream impact to Microsoft 365 services has been communicated via MO1437424. What went wrong and why? Azure datacenters connect through redundant network paths, ensuring that traffic can continue flowing even when one path is unavailable for maintenance or repair. When unplanned maintenance – including ‘break-fix’ hardware repair – is needed on an optical device along one of these paths, the repair process starts by creating system-readable instructions, verifying that the redundant path remains healthy, and performing safety checks to confirm that the work will not affect customer traffic. Because break-fix activities on a single device are generally expected to be impactless, these requests are processed automatically without human engagement or triage. On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk. A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter. The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe. The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter's worth of devices in a single request. As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region. Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption. The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure. Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy. Meanwhile, strong signals indicating WAN impact drew much of the investigative focus. Two parallel investigation streams developed: one focused on the WAN routing anomaly, and another examining network device health from within the datacenter. It took additional time to work past these misleading signals and eventually identify the break-fix activity as the source. It was not until 17:19 UTC that our engineers were able to correlate the routing anomaly with the break-fix activity, at which point we had identified the trigger event and began planning a rollback. Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful. After our engineers correlated the routing anomaly with the break-fix activity, we identified that the incomplete preparation activities established the trigger conditions. We initiated a manual rollback at 17:45 UTC, then datacenter-to-WAN connectivity was restored when the rollback completed at 18:26 UTC, and all impacted services fully recovered by 19:41 UTC. How did we respond? • 14:44 UTC on 23 July 2026 – Device break-fix began. Earliest potential customer impact time. • 14:45 UTC on 23 July 2026 – Service alerts fired and multiple Azure service teams started investigating the traffic anomalies. • 14:54 UTC on 23 July 2026 – Automated recovery system started performing multiple retries to recover the devices that were affected by the over scoping that happened during the break-fix issue. • 15:26 UTC on 23 July 2026 – Routing anomaly alert incorrectly triaged to WAN issue and was sent to engineers to investigate further. • 16:58 UTC on 23 July 2026 – After multiple retries, automated recovery system raised an alert to engineers to investigate. Engineers tried to recover the devices manually. • 17:19 UTC on 23 July 2026 – Engineers correlated the routing anomaly with the initial break-fix trigger event and began attempting auto-rollback. • 17:45 UTC on 23 July 2026 – Engineers initiated manual rollback of the break-fix. • 18:26 UTC on 23 July 2026 – Rollback was successful on all affected devices, engineers continued monitoring recovery of affected services. • 19:41 UTC on 23 July 2026 – Customer impact confirmed as mitigated, after all services returned to expected operating levels. How are we making incidents like this less likely or less impactful? • We have fixed the defect in the blast radius analysis system that incorrectly expanded the scope of this ‘break-fix’ unplanned maintenance to include additional devices. Input validation now ensures that both diverse paths for a datacenter cannot undergo break-fix requests at the same time. (Completed) • We have scanned all existing break-fix requests in the system to identify similar conditions that could have allowed the same defect to produce similar impact elsewhere. (Completed) • We have enhanced safety checks to block any break-fix or maintenance activity that would affect all diverse paths for a datacenter, to de-risk this failure mode. (Completed) • We are improving change visibility, break-fix requests will be recorded in the standard Azure change ledger, to reduce the time required to correlate maintenance activity with network issues during future investigations. (Estimated completion: August 2026) • We are enhancing our datacenter traffic monitoring to detect abnormalities in inbound and outbound traffic more quickly, and surface these directly to engineers for evaluation – enabling faster customer notification, and mitigation efforts. (Estimated completion: September 2026) • We are improving change tooling guardrails so that multiple devices cannot be isolated simultaneously during maintenance. (Estimated completion: September 2026) • In the longer term, we are improving our automated recovery system to fail more quickly and escalate to engineers more promptly when retries are unsuccessful, rather than continuing to retry against disrupted connectivity. (Estimated completion: October 2026) • Finally, we are improving our automated rollback so it can recover more robustly, even when the underlying connectivity on which it depends is degraded. (Estimated completion: October 2026) How can customers make incidents like this less impactful? • For mission-critical workloads, customers should consider a multi-region deployment strategy to maintain availability during events that affect a single region: https://learn.microsoft.com/azure/architecture/patterns/geodes and https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources. For guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/ZJV6-SGG

Read the full incident report →

Started Jul 08, 2026, 07:47 PM UTC· resolved Jul 08, 2026, 09:52 PM UTC

1. monitoring Jul 08, 2026, 07:47 PM UTC

   GitHub Enterprise Cloud with Data Residency is experiencing a service outage impacting all services, issue identified and mitigation in progress. For details please go here: : https://us.githubstatus.com/posts/details/PBJ0QTY

Read the full incident report →

Started Jun 15, 2026, 11:44 PM UTC· resolved Jun 16, 2026, 12:10 AM UTC

1. monitoring Jun 15, 2026, 11:44 PM UTC

   Impact Statement: Starting at 21:45 UTC on 15 June 2026, a subset of Azure customers leveraging services which depend upon MCR (Microsoft Container Registry) for storing images may experience 401 authentication errors for requests attempting to pull Microsoft images from mcr.microsoft.com.Current Status: We detected this issue via internal service monitoring upon identifying a spike of 401 errors. We’ve identified a potential underlying factor to be a recent update which introduced a code regression, and we've completed roll out of a hotfix to address the code regression and mitigate impact for affected customers. We are closely monitoring internal telemetry to validate full mitigation and prevent further occurrences. The next update will be provided within 60 minutes, or as events warrant.

2. resolved Jun 16, 2026, 12:01 AM UTC

   What happened?Between 21:45 UTC and 23:23 UTC on 15 June 2026, Azure customers leveraging services which depend upon Microsoft Artifact Registry (MAR) may have experienced 401 authentication errors when attempting to pull images from MAR . This incident is now mitigated. Further details will be provided directly to impacted customers through the Azure Portal.

Read the full incident report →

Started Jun 15, 2026, 11:04 PM UTC· resolved Jun 15, 2026, 11:50 PM UTC

1. monitoring Jun 15, 2026, 11:04 PM UTC

   We are actively investigating customer alerts of 401 authentication errors when pulling images from mcr.microsoft.com. We’ve identified a potential underlying factor related to a recent update made to a backed service and we are actively working on determining mitigation workstreams. More information will be provided shortly.

Read the full incident report →

Started May 29, 2026, 09:39 AM UTC· resolved May 29, 2026, 12:58 PM UTC

1. monitoring May 29, 2026, 09:39 AM UTC

   Impact Statement: Starting at 09:39 UTC on 29 May 2026, you have been identified as a subset of customers using Azure OpenAI service in multiple regions who may experience degraded request success where some requests returned errors or failed to complete. Current Status: We identified this issue through internal automated service monitoring after detecting a drop in call success rates. Our investigation indicates that the issue is linked to elevated server-side errors within a backend service layer in the affected region. We are actively investigating and working to determine the full scope and mitigation steps. The next update will be provided within 60 minutes, or sooner if there are significant developments.

2. postmortem May 29, 2026, 05:05 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/LYXT-C1Z What happened? Between 09:39 UTC and 17:05 UTC on 29 May 2026, customers using the Azure OpenAI Service may have experienced increased latency, intermittent request failures, timeouts, and/or HTTP 5XX errors when submitting inference requests. The impact was more pronounced in regions in Europe and Australia East that were directly processing the highest levels of affected traffic. The incident was triggered by an upstream change that altered how certain capacity-related failures were surfaced, which led to a rapid and unexpected increase in internal retry traffic. This retry amplification overwhelmed a shared inference load balancer component, which distributes the inference traffic, degrading service reliability across multiple regions. What went wrong and why? Azure OpenAI relies on an internal routing and load balancing layer to direct inference requests across regions and model capacity. This layer is designed to distribute traffic efficiently and support resiliency during transient failures. During this incident, that layer became overloaded and unstable after it was subjected to a large volume of retry traffic from an upstream internal workload. The triggering condition was a change in an upstream API layer used by a Microsoft first-party workload, Microsoft 365, which relies on Azure OpenAI for some request processing scenarios, including Copilot-related experiences. This change altered how capacity-related failures were surfaced to higher layers in the request path. The rollout first occurred in Australia East, and later expanded to Sweden Central. Before this change, some lower-level capacity or resource exhaustion conditions could still result in retry activity, as designed - but after repeated failures, they would ultimately return in a form that did not trigger broad retry behavior across multiple upstream layers. After the change, those same conditions began surfacing as transient server failures instead. Because those responses were now interpreted as retriable, multiple layers in the calling stack retried immediately and repeatedly, without sufficient backoff or jitter. In some cases, a single failed request resulted in up to 48 additional retry attempts. This created a cascading retry amplification effect that dramatically increased internal traffic volume, even though underlying customer demand had not materially changed. The first wave of impact occurred in Australia East, where the change was initially deployed. As retry traffic increased, it began exhausting resources in the Azure OpenAI inference routing layer and triggered failures in the affected backend path. During that first wave, early crash signatures pointed to an internal feature associated with error handling, which appeared to be a plausible explanation for the instability. We disabled that feature as a potential mitigation, and around the same time traffic in Australia East naturally declined as part of the normal regional usage pattern. Because service health improved after those two things happened in close succession, it initially appeared that the mitigation had addressed the problem. In reality, the feature we disabled was not the underlying cause, and the retry amplification issue remained in place. When the upstream rollout later expanded to Sweden Central, the same failure pattern reappeared, but at much greater scale. As a major European processing hub, Sweden Central serves significantly higher traffic volume - so the retry amplification effect was stronger and the resulting overload was more severe. This second wave made it clear that the issue was not limited to the internal feature we had first suspected. Instead, it exposed the true underlying problem: a large internal retry storm overwhelming the shared routing layer. As the retry storm intensified, it exhausted resources in the Azure OpenAI inference routing layer. The affected component scaled out, but the magnitude of the retry amplification exceeded the protection available at the time. In a small number of regions, recovery was further constrained because scale-out options were limited by available compute quota for replacement virtual machine SKUs. The result was resource exhaustion, out-of-memory crashes on multiple instances, and broad degradation across the shared routing path. This incident also exposed a gap in how internal workloads were protected, relative to traffic from external sources. The affected upstream workload operated in a model where traffic was not subject to the same rate-limiting and overload controls used for many external scenarios. In addition, some of this traffic reached the internal routing layer more directly than standard external traffic paths. That meant the system did not sufficiently suppress or isolate the amplified retry volume before it reached shared infrastructure. While the initial impact in Australia East led us toward a credible but incomplete explanation, the apparent recovery after disabling the suspected feature obscured the real cause. Once the rollout reached Sweden Central and the issue reappeared, albeit at greater scale, we were able to definitively correlate the degradation to retry amplification, correctly diagnose the trigger condition as the change in the upstream dependency API layer, then execute the targeted mitigations that restored stability. How did we respond? • 09:20 UTC on 29 May 2026 – Initial customer impact began as the first wave of retry amplification, associated with the upstream rollout in Australia East, caused increased failures and latency across multiple regions. • 09:39 UTC on 29 May 2026 – Automated monitoring detected a drop in request success rates and initiated incident investigation. • 10:10 UTC on 29 May 2026 – We identified widespread service instability and correlated failures across regions and models. • 12:17 UTC on 29 May 2026 – Based on early crash diagnostics, we disabled a component that was suspected of contributing to the instability. • 12:40 UTC on 29 May 2026 – Service health began improving as traffic conditions in Australia East eased, which initially made the earlier mitigation appear effective. • 14:20 UTC on 29 May 2026 – The upstream rollout expanded to Sweden Central, where significantly higher traffic volume increased the impact of the retry amplification. • 14:40 UTC on 29 May 2026 – The issue reoccurred after the upstream rollout reached Sweden Central, where higher traffic volume magnified the retry amplification and overloaded the system at greater scale. • 16:20 UTC on 29 May 2026 – We identified that the earlier mitigation had addressed a secondary symptom rather than the underlying cause, and refocused the investigation on upstream retry behavior. • 16:27 UTC on 29 May 2026 – We confirmed that the amplified traffic was originating from an internal first-party workload and began targeted mitigation planning. • 16:30 UTC on 29 May 2026 – We identified the source of the amplified retry traffic, isolated the offending internal workload onto dedicated infrastructure, and coordinated with the upstream service to roll back the triggering change and reduce traffic volume. • 16:25 UTC on 29 May 2026 – The upstream service began rollback of the triggering change and took additional actions to reduce retry-driven traffic volume. • 17:05 UTC on 29 May 2026 – Following the completion of the isolation and rollback, and all backlogged requests were cleared, customer impact was confirmed as mitigated. How are we making incidents like this less likely or less impactful? • We are reducing shared infrastructure blast radius by moving large first-party generative AI workloads onto dedicated routing infrastructure, rather than allowing them to share the same inference load balancing stamp with broader multi-tenant traffic. (Estimated completion: June 2026) • We are improving our diagnostic guidance, runbooks, and alerting so that engineers can more quickly distinguish between secondary crash symptoms and the true underlying source of overload during fast-moving incidents. (Estimated completion: June 2026) • We are partnering with upstream service teams to improve retry policy design, dependency contracts, and telemetry so retry amplification conditions can be detected earlier and prevented from propagating across shared infrastructure. (Estimated completion: July 2026) • We are implementing stronger overload prevention and workload throttling controls so that excessive retry traffic from any single internal workload cannot overwhelm shared components. (Estimated completion: July 2026) • Finally, we are continuing work to identify and eliminate single points of failure and improve resiliency in the inference routing layer. (Perpetual/Ongoing) How can customers make incidents like this less impactful? • For mission-critical AI workloads, customers should consider a multi-region resiliency strategy so that service degradation affecting a specific processing path or region is less likely to impact the full application experience. Customers using regional deployments may wish to evaluate whether additional geographic diversity or fallback patterns would improve resilience for their scenarios. See: https://learn.microsoft.com/azure/ai-services/openai/how-to/business-continuity-disaster-recovery • Customers should also review application retry behavior to ensure that client-side retries use appropriate backoff and jitter, and that applications degrade gracefully when dependent services are under stress. Well-controlled retry behavior helps avoid compounding platform-level recovery events. See: https://learn.microsoft.com/azure/architecture/patterns/retry • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/LYXT-C1Z

Read the full incident report →

Started May 29, 2026, 07:07 AM UTC· resolved May 29, 2026, 07:42 AM UTC

1. monitoring May 29, 2026, 06:57 AM UTC

   Impact Statement- Starting at 04:27 UTC on 29 May 2026, customers may experience increased latency, intermittent connectivity, or timeouts when accessing resources hosted in the West US 2 region. Engineers are actively investigating the issue and working to restore network stability. Some requests may succeed upon retry as connectivity intermittently recovers. We are sending targeted communications to impacted customers through the Azure Service Health dashboard. The next update will be provided within 60 minutes.

2. monitoring May 29, 2026, 07:07 AM UTC

   Impact Statement- Starting at 04:27 UTC on 29 May 2026, customers may experience increased latency, intermittent connectivity, or timeouts when accessing resources hosted in the West US 2 region, due to a power event. We are in process of recovering power and working to restore infrastructure in the region. We started seeing partial recovery and will be sending targeted communications to impacted customers through the Azure Service Health dashboard. The next update will be provided within 60 minutes.

Read the full incident report →

Started May 29, 2026, 04:24 AM UTC· resolved May 29, 2026, 09:45 PM UTC

1. monitoring May 29, 2026, 04:27 AM UTC

   Impact Statement: Starting at 04:27 UTC on 29 May 2026, we are investigating an incident which is impacting multiple services in West US 2, due to a power event. This impacts service accessibility in the region, with customers potentially experiencing increased latency and intermittent connectivity, including timeouts when connecting to resources. Current Status: We identified this issue through our automated monitoring after detecting elevated alerts related to network and infrastructure health in the West US 2 region. The issue was triggered by a power event in the datacenter, which impacted underlying infrastructure and led to increased latency and intermittent connectivity for select resources. We are aware of the impact and are actively investigating. Mitigation efforts are underway to restore affected network components across dependent services, including Storage and Compute. Customers may begin to see signs of recovery as these actions progress. The next update will be provided within 60 minutes, or sooner if there are significant developments.

2. monitoring May 29, 2026, 04:27 AM UTC

   Impact Statement: Starting at 04:27 UTC on 29 May 2026, a severe thunderstorm caused widespread utility power loss to our West US 2 datacenter facilities, resulting in a multi-service outage. Impact is currently region-wide, and we have not yet confirmed isolation to specific scale units or availability zones and are treating this as full-region impact until physical inspections confirm otherwise. Customers are experiencing service connectivity failures, timeouts, and elevated error rates across affected services. This includes an inability to deploy new resources or scale existing workloads, intermittent availability where some requests may succeed while others fail depending on which infrastructure nodes have recovered, and both data plane and control plane impact for services not yet restored. Affected Azure services include, but are not limited to:Azure Functions, Azure Database for MySQL Flexible Servers, Azure Database for PostgreSQL Flexible Server, Azure Databricks, Redis, Azure SQL, Azure Managed Grafana, Virtual Machines, Virtual Machine Scale Sets, Azure Kubernetes Service, Storage, Application Insights, Azure Data Factory, Azure Monitor, and Azure Log Analytics Current Status: Severe thunderstorms resulted to a complete utility power loss to multiple datacenter buildings simultaneously. Our backup generators activated but were unable to fully compensate. The generator transfer switchover was only partially successful, with some generator sets failing to synchronize under the sudden full facility load and others experiencing thermal protection shutdowns as ambient temperatures rose due to simultaneous cooling system failures. This cascading combination of power instability and cooling loss exceeded our designed N+1 redundancy for this failure mode. We are still determining the full extent of mechanical vs. electrical failures in the generator systems. HVAC systems are being restarted in sequence; ambient temperatures are returning to safe ranges but have not fully normalized, so we are staged-powering equipment to avoid thermal re-trips. Network devices and storage arrays are being power-cycled and validated in sequence, with storage dependencies as the primary remaining bottleneck due to required manual intervention and data integrity verification. Where possible, traffic is being redirected to healthy nodes, and on-site teams are conducting physical hardware inspections to identify components requiring replacement. The following services have been confirmed restored and are operating normally:Service Bus, App Service (Web Apps), Azure Site Recovery, Backup (MAB), Azure Cosmos DB, Azure Resource Manager, Data Explorer, Azure IoT Hub, Microsoft Defender for Cloud Apps, Azure Container Registry, Azure Policy, Azure NetApp Files, Azure Resource Graph, Azure Synapse Customer Guidance: • If you have resources in paired or alternate regions, consider failing over traffic away from West US 2 until we confirm full recovery. • Single-region workloads will recover automatically as infrastructure is restored — no customer action is required. • We recommend pausing new deployments to West US 2 until this incident is resolved; use alternate regions if urgent. • Monitor Azure Status (status.azure.com) and Service Health in the portal for real-time updates specific to your subscriptions. Next update: We will provide our next status update within 1 hour, or sooner, if events warrant.

3. monitoring May 29, 2026, 04:27 AM UTC

   Impact statement: Starting at 04:27 UTC on 29 May 2026, a severe thunderstorm caused widespread utility power loss across multiple West US 2 datacenter facilities, resulting in impact to multiple Azure services. Datacenter power has been fully restored, and the majority of network and storage infrastructure has recovered. Infrastructure in Availability Zone 2 has recovered and is operating normally. Residual impact is now concentrated on a subset of infrastructure within Availability Zones 1 and 3, where two storage stamps are undergoing final recovery and validation. Customers with resources deployed exclusively in Availability Zone 2 should not be affected by the remaining recovery activities; however, those with dependencies in Availability Zones 1 or 3 may continue to experience intermittent connectivity, elevated latency, or resource unavailability. Currently affected Azure services include, but may not be limited to (recovery validation underway):Azure Functions, Azure Database for MySQL Flexible Server, Azure Database for PostgreSQL Flexible Server, Azure SQL, Azure Managed Grafana, Virtual Machines, Virtual Machine Scale Sets, Azure Kubernetes Service, Azure Storage, Application Insights, Azure Monitor, and Azure Log Analytics.The following services have recovered and are operating normally:Service Bus, App Service (Web Apps), Azure Site Recovery, Backup (MAB), Azure Cosmos DB, Azure Resource Manager, Azure Data Explorer, Azure IoT Hub, Microsoft Defender for Cloud Apps, Azure Container Registry, Azure Policy, Azure NetApp Files, Azure Resource Graph, Azure Data Factory, Azure Databricks, Azure Cache for Redis, and Azure Synapse Analytics. Current Status: A severe weather event, including multiple lightning strikes, affected several datacenter facilities in the West US 2 region, resulting in concurrent utility power interruptions across multiple availability zones. Backup generators activated as designed; however, during the transition to sustained generator operation, a subset of generator systems were unable to fully synchronize under the sudden facility load, while others subsequently shut down due to thermal protection mechanisms as cooling systems were impacted by the broader power disruption. Because these events occurred across multiple facilities within the same timeframe, the resulting conditions exceeded the resiliency designed for this particular failure scenario. Datacenter utility power has been fully restored. HVAC systems have normalized, ambient temperatures have returned to expected operating ranges, and the majority of network and storage infrastructure has recovered successfully. While most services have returned to normal operation, a small subset of storage clusters and network devices did not recover cleanly and continue to limit full-service restoration. Some storage clusters remain partially unavailable or dependent on ongoing network recovery activities, and a limited number of network devices connecting compute resources to storage infrastructure remain impaired. As a result, some virtual machines are currently unable to access their underlying storage dependencies and may remain unavailable. Recovery efforts have transitioned from broad infrastructure restoration to targeted remediation of specific network devices, storage clusters, and isolated compute nodes. As these components are restored, validated, and returned to service, dependent workloads are recovering incrementally. Two storage stamps are currently completing final recovery and integrity validation activities, and services dependent on those stamps are expected to recover as this work is completed. Estimated Time to Resolution: With datacenter power restored and the majority of network and storage infrastructure recovered, remaining work is focused on two storage stamps completing final recovery and data integrity validation activities. Based on current progress, we estimate full-service restoration within approximately 2-3 hours. Individual services dependent on these stamps will recover progressively as validation activities are completed. Customer Guidance:Single-region workloads will recover automatically as infrastructure restoration activities are completed; no customer action is required.We recommend postponing new deployments into West US 2 until this incident is resolved and using alternate regions for urgent deployment needs.Logical availability zones assigned to customer subscriptions may map to different physical availability zones. Customers can use the Locations API to understand this mapping: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings. Next Update: We will provide our next status update within 60 minutes, or sooner if significant progress is made.

4. postmortem May 30, 2026, 02:30 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/GHRP-84G What happened? Between 04:24 UTC on 29 May and 02:30 UTC on 30 May 2026, customers with resources in the West US 2 region may have experienced failures when attempting to access or manage some Azure services in the region. Impact included connectivity failures, resource unavailability, and service management failures. The issue was triggered by a severe thunderstorm that caused utility power disturbances affecting multiple datacenter facilities serving the region. These disturbances led to some cooling systems entering a protective mode to preserve equipment integrity. This caused elevated temperatures which triggered proactive shutdowns of infrastructure affecting compute, networking, and storage systems across two different availability zones in the region (physical AZ-01 and AZ-03). After cooling systems stabilized, storage services - and the networking on which they depend - required extensive manual recovery actions, which contributed to the total duration of customer impact. The following services were among those affected: Application Insights, Azure App Service, Azure Backup, Azure Cache for Redis, Azure Container Registry, Azure Cosmos DB, Azure Databricks, Azure Data Explorer, Azure Data Factory, Azure Functions, Azure IoT Hub, Azure Kubernetes Service, Azure Monitor, Azure NetApp Files, Azure Resource Manager, Azure Site Recovery, Azure SQL Database, Azure Storage, Azure Synapse Analytics, Azure Virtual Machines, Log Analytics, Service Bus, and Virtual Machine Scale Sets. What went wrong and why? A severe thunderstorm produced lightning strikes, resulting in utility power disturbances across multiple datacenter facilities serving the West US 2 region. These disturbances affected a wide enough geographic area that a subset of datacenters within two Availability Zones experienced impact. While this event was not a complete utility power interruption, these datacenters experienced multiple voltage sag/swell events, meaning utility power remained present but was unstable. The region consists of datacenters built at different times with varying infrastructure designs, including differences in cooling control systems and power protection mechanisms. Some datacenters transferred to generator power, while others did not transfer, as the utility power source remained available even as voltage fluctuated. Independently of whether each datacenter transferred to backup power, a subset of components within the mechanical cooling systems - across multiple datacenters - detected abnormal electrical conditions and, by design, entered a 'lockout' protective state. This reduced cooling capability and prevented parts of the cooling system from automatically returning to normal operation after the power disturbance. As temperatures rose beyond safe operating thresholds, a subset of cloud infrastructure automatically shut down to prevent physical damage and preserve data integrity. Impacted cooling systems required manual intervention and troubleshooting, including multiple restart attempts before stable cooling capacity could be reestablished. Additionally, we had to identify which devices required manual recovery and which were expected to self-recover. Impacted Storage scale units required additional validation. These factors contributed to the extended recovery time for services dependent on storage, including telemetry processing systems. How did we respond? Recovery efforts began immediately, prioritizing cooling restoration, then network infrastructure, followed by compute and storage systems. Cooling was restored within approximately two hours of impact start. The majority of compute resources recovered within eight hours. Storage validation, which must be performed sequentially, extended over approximately 14 hours. Finally, telemetry services (Application Insights and Log Analytics) required an additional six hours beyond storage recovery to process accumulated backlogs. The timeline below details the progression of recovery: • 04:14 UTC on 29 May 2026 – Utility power sag/swells were observed across multiple datacenters. These caused a subset of the mechanical cooling systems to enter a 'lockout' protective state. • 04:24 UTC on 29 May 2026 – Customer impact began; our monitoring produced thermal alerts as temperatures rose due to reduced cooling functionality. • 04:31 UTC on 29 May 2026 – Manual cooling diagnosis and restoration efforts began. • 04:40 UTC on 29 May 2026 – As temperatures rose, some cloud infrastructure shut down to prevent damage. • 04:50 UTC on 29 May 2026 – Networking and Storage engineers began assessing infrastructure health and necessary restoration efforts. • 05:55 UTC on 29 May 2026 – All cooling was restored in impacted datacenters, and temperatures had stabilized. Assessments continued on remaining infrastructure issues. • 06:15 UTC on 29 May 2026 – Approximately 50% of impacted underlying Virtual Machines (VMs) had recovered. • 12:00 UTC on 29 May 2026 – Approximately 95% of impacted underlying Virtual Machines (VMs) had recovered, after premium Storage had been largely recovered. Compute and Storage teams continued to identify and mitigate remaining unhealthy nodes. • 18:46 UTC on 29 May 2026 – Storage was confirmed as the remaining recovery bottleneck; validation of Storage-specific issues was ongoing. • 20:18 UTC on 29 May 2026 – All affected services confirmed as recovered, except for Application Insights and Log Analytics which were still processing backlogs. • 02:30 UTC on 30 May 2026 – Customer impact for Log Analytics and Application Insights confirmed as mitigated. How are we making incidents like this less likely or less impactful? • First and foremost, we are reviewing and enhancing our operational processes for cooling system recovery based on learnings from this incident, to ensure that all employees have clear and documented actions to support faster recovery during high-stress scenarios. (Estimated completion: June 2026) • Our Log Analytics team is improving the resiliency of service startup processes so that transient initialization failures self-recover automatically, reducing the need for manual intervention during large-scale restarts. (Estimated completion: June 2026) • Our Networking team is addressing platform limitations that constrained the speed at which storage nodes could be returned to service following the thermal event. This includes reducing dependencies on manual coordination steps, and enabling faster parallel recovery of storage infrastructure. (Estimated completion: July 2026) • Our Networking team is also improving the tooling and processes used to identify which infrastructure devices have entered protective states and require manual recovery. This includes developing clearer prioritization of foundational devices (such as management and networking components) earlier in the recovery sequence, and improving awareness across teams when thermal events affect shared infrastructure. (Estimated completion: August 2026) • Our Log Analytics team is improving the architectural resiliency of our telemetry processing services by enabling multi-cluster flexibility and support for a broader pool of compute resource types, reducing dependence on any single cluster or resource type's availability and improving the ability to recover during large-scale scenarios. (Estimated completion: August 2026) • Finally, in the longer term we are performing a joint, holistic engineering review to evaluate the fail-state behavior of the cooling system protection logic. The current system prioritizes equipment protection and data security over operational continuity, which resulted in a full lockout rather than staged or degraded operation. This includes evaluating proactive row-level shutdown strategies during rapid temperature increases, and expanding the decision-making window available to operators before protective systems engage. (Estimated completion: October 2026) How can customers make incidents like this less impactful? • Note that the 'logical' Availability Zones used by each customer subscription may correspond to different physical Availability Zones - customers can use the Locations API to understand this mapping, to confirm which resources run in which physical AZ: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/en-us/azure/architecture/patterns/geodes and https://learn.microsoft.com/en-us/azure/well-architected/design-guides/regions-availability-zones • Consider leveraging data redundancy options such as Geo-Redundant Storage (GRS) or Read-Access Geo-Redundant Storage (RA-GRS), which replicate data to a secondary region and can provide continued read access to data during regional incidents: https://learn.microsoft.com/azure/storage/common/storage-redundancy • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources. For guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/GHRP-84G

Read the full incident report →

Started May 27, 2026, 01:01 PM UTC· resolved May 27, 2026, 02:44 PM UTC

1. monitoring May 27, 2026, 01:01 PM UTC

   Impact: From 08:40 UTC, 27 May 2026, some customers using Azure Synapse Analytics (private link, East US) may experience delays or failures accessing Synapse Studio or workspaces. We detected elevated connection failures via automated monitoring and are actively investigating.

2. monitoring May 27, 2026, 02:05 PM UTC

   Impact: From 08:40 UTC, 27 May 2026, some customers using Azure Synapse Analytics (private link) may experience delays or failures accessing Synapse Studio or workspaces across multiple regions. As of now mitigation is in progress, and we are seeing initial signs of recovery.

Read the full incident report →

Started May 18, 2026, 07:22 PM UTC· resolved May 18, 2026, 10:17 PM UTC

1. monitoring May 18, 2026, 05:24 PM UTC

   Impact Statement: Starting from 13:59 UTC on 18 May 2026, customers using Microsoft Fabric - including Power BI and PowerBI Embedded - may be experiencing delays or failures while interacting with the service. This issue is affecting multiple regions in the Azure public cloud. Current Status: We are aware of this issue and are actively investigating. Our analysis indicates that an unexpected increase in request traffic is causing a portion of the infrastructure to operate below optimal levels. Our on-call engineering teams are investigating relevant mitigations. There is no workaround available at this time. We expect to be able to resolve this issue by 18:45 UTC, approximately 90 minutes from now. The next update will be provided in 60 minutes, or as events warrant.

2. monitoring May 18, 2026, 06:33 PM UTC

   Impact Statement: Starting from 13:59 UTC on 18 May 2026, customers using Microsoft Fabric - including Power BI and PowerBI Embedded - may be experiencing delays or failures while interacting with the service. This issue is affecting multiple regions in the Azure public cloud. The Microsoft Fabric service status page is being updated at: https://support.fabric.microsoft.com/support/. Current Status: We identified that an unexpected increase in request traffic is causing a portion of the infrastructure to operate below optimal levels. Our on-call engineering teams have identified what we believe to be the relevant contributing factors, and have started to apply mitigations. We are initially applying these in a single region to help validate their effectiveness, before applying them more broadly. We are not able to provide a revised estimate of when this issue will be resolved - but the next update will be provided by 19:30 UTC (in approximately 60 minutes) or sooner if we have more to share.

3. monitoring May 18, 2026, 07:22 PM UTC

   Impact Statement: Starting from 13:59 UTC on 18 May 2026, customers using Microsoft Fabric - including Power BI and PowerBI Embedded - may be experiencing delays or failures while interacting with the service. This issue is affecting multiple regions in the Azure public cloud. Current Status: For the latest details, please refer to the Microsoft Fabric service status page: https://support.fabric.microsoft.com/support/

Read the full incident report →

Started Apr 24, 2026, 11:30 AM UTC· resolved Apr 24, 2026, 11:22 PM UTC

1. monitoring Apr 24, 2026, 11:39 AM UTC

   Impact Statement: Starting at 11:39 UTC on 24 April 2026, a platform issue resulted in provisioning, scaling, and connectivity issues for customers with Azure workloads hosted in the East US region. This issue initially affected one availability zone in East US. As traffic was automatically rerouted and shared services were involved, some customers in other availability zones may have also experienced an impact. Current Status: We detected this issue through our automated monitoring, which identified an unusual increase in failures within the services supporting virtual machine and virtual machine scale set operations in the East US region. We have taken steps to stabilize the affected services and address the underlying issue, while continuing the investigation to determine contributing factors. The next update will be shared within 60 minutes, or as events warrant.

2. postmortem Apr 24, 2026, 11:22 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/5GP8-W0G What happened? Between 11:30 and 23:22 UTC on 24 April 2026, customers may have experienced failures or delays when attempting to provision, scale, or update resources in East US. Beyond this impact to service management, a small subset of customers may have encountered intermittent connectivity issues for newly-provisioned workloads including Virtual Machines and Azure Virtual Desktop sessions. The issue initially began with impact to a subset of customers in a single Availability Zone (physical AZ-01) but as demand shifted, similar symptoms were observed impacting service management operations in AZ-02 and AZ-03. While none of these zones were impacted for the full duration of the incident, customers could have experienced periods of impact in each zone for portions of the incident. The following services were among those affected: Azure Application Gateway, Azure App Service, Azure Batch, Azure Cache for Redis, Azure Data Explorer, Azure Data Factory, Azure Databricks, Azure Health Data Services, Azure Kubernetes Service (AKS), Azure Red Hat OpenShift, Azure Service Fabric, Azure Synapse Analytics, Azure Virtual Desktop, Azure Virtual Machines (VMs), Azure Virtual Network Manager, Azure VMware Solution, Oracle Database@Azure, Virtual Machine Scale Sets – and potentially additional services that were dependent on new compute allocations in the region. Note: Logical availability zones assigned to customer subscriptions may map to different physical availability zones. Customers can use the Locations API to understand this mapping: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings. What went wrong and why? The Azure PubSub service is a key component of the networking control plane, acting as an intermediary between resource providers and networking agents on Azure hosts. Resource providers, such as the Network Resource Provider (NRP), publish customer configurations during VM or networking ‘create, update, or delete’ operations. Networking agents (subscribers) on the hosts retrieve these configurations from the resource providers to program the host's networking stack. Additionally, the PubSub service functions as a cache, ensuring efficient retrieval of configurations during the lifecycle of a VM. This capability is essential for resource allocation and network mappings in Azure Virtual Network (VNet) environments. Like many Azure control plane services, Azure PubSub leverages Azure Service Fabric for distributed state management through a compute and data co-locating architecture. This is designed to enable low-latency, high-performance execution with built-in consistency by keeping application logic and replicated state together on the same nodes. During normal platform operations, one partition of the PubSub control plane service in a single Availability Zone (AZ-01) experienced lock contention errors for several minutes, resulting in operations on that partition to timeout and fail. As designed, the platform initiated an automatic failover to a secondary replica, but this failover did not complete successfully. We intervened to investigate and attempted a manual failover of the primary partition, but this attempt was also unsuccessful, therefore resources on the impacted partition continued to see service management failures. Approximately two hours into troubleshooting, we received alerts of impact to a single partition in AZ-03, which indicated partial loss of control plane availability in this zone as well. As the investigation progressed, we suspected that a previously deployed update to a control plane dependency had introduced a regression which could trigger lock contention under certain conditions.As part of our mitigation efforts, we prepared a rollback to the ‘last known good’ version to eliminate the possibility of a regression. We first applied this rollback in AZ-03, which successfully restored control plane service in that zone. Based on this, we began rolling back the affected components in AZ-01. By design, rollback operations are executed in stages by our Service Fabric controllers using update domains to ensure platform safety. Recovery time was further extended because compute and data are co-located on the same nodes. In some cases, rollback requires rebuilding full replicas on new nodes, which significantly increases the time required to complete each stage. Given these conditions, we intentionally prioritized maintaining availability for unaffected partitions and services rather than accelerating the rollback. This combination of sequential update domain processing, replica rebuild requirements, and our standard safe recovery approach, resulted in an extended time to mitigate. While mitigation was in progress, there were two periods of time during which we were unable to maintain two simultaneous fully healthy instances of the PubSub service across availability zones, a requirement for normal replication and control plane operations. This resulted in a temporary loss of quorum within the service. As the service attempted to self-heal, customer impact shifted between availability zones, leading to periods of degraded behavior across multiple zones. These failure patterns began to appear in AZ-02 and again in AZ-03, broadening the scope of impact across the region. For AZ-02 we initiated and completed a rollback, and although AZ-03 had previously shown recovery following the earlier attempted rollback, we discovered that the rollback in that zone had not fully completed to all update domains. As impact reemerged, a rollback in AZ-03 was reinitialized and then completed, fully restoring service health. Our post incident investigations confirmed that a latent regression in a recently deployed version of the PubSub service exacerbated the replica build times for the large Availability Zones of this region. This regression also contributed to the failure of the failover process, as it increased contention and reduced the service’s ability to build replicas under these conditions successfully. This issue had not surfaced during earlier testing, or in any other regions which had already received the update. During the early stages of the incident, the impact appeared limited to a specific set of VM service management operations. Our first customer communication was automatically sent to impacted customers within 30 minutes of the issue starting, via our “BRAIN” automated messaging system. As more signals surfaced, it became clear that the impact was becoming broader and affecting the underlying systems on which multiple services rely, across Availability Zones. Consequently, we expanded our communications from attempting to target affected subscriptions, to communicating to all customers with resources in the region – in addition to publishing to our public Azure Status page. Finally, some updates lacked specificity that could have helped customers understand the evolving impact, how widespread the issue was, and how best to respond. We have included communications-related learnings below, to improve both the speed and level of detail. How did we respond? • 11:30 UTC on 24 April 2026 – Customer impact began, including failures and delays when attempting to provision, scale, or update resources. • 11:38 UTC on 24 April 2026 – We detected an issue in AZ-01. A control plane partition experienced lock contention, and automatic failover attempts to secondary replicas did not complete successfully. • 11:38–13:40 UTC on 24 April 2026 – We attempted manual failover in AZ-01, but these efforts did not successfully restore service. • 11:59 UTC on 24 April 2026 – First targeted notifications were sent to a subset of impacted customers, via our “BRAIN” automated messaging system. • 13:14 on 24 April 2026 – Notifications expanded to customer subscriptions that were using VMs/VMSS resources in the region. • 13:40 UTC on 24 April 2026 – We identified a recently deployed update as the likely trigger condition for this issue. • 13:50 UTC on 24 April 2026 – We began observing impact in AZ-03, indicating the issue was now affecting multiple Availability Zones. • 14:07 UTC on 24 April 2026 – We initiated a rollback to the last known good version in AZ-03. • 15:03 UTC on 24 April 2026 – We observed significant recovery in AZ-03, control plane availability exceeded 99%. • 15:04 UTC on 24 April 2026 – We initiated a rollback to the last known good version in AZ-01. • 15:36 UTC on 24 April 2026 – First public Azure Status page update, to ensure wider visibility. • 17:19 on 24 April 2026 – Notifications expanded to all customer subscriptions that were using any resources in the region. • 18:52 UTC on 24 April 2026 – We observed significant improvement in AZ-01, as the rollback progressed. • 19:02 UTC on 24 April 2026 – We observed significant recovery in AZ-01, control plane availability exceeded 99%. • 19:05 UTC on 24 April 2026 – We began observing impact in AZ-02, as load redistributed across the region. • 19:10 UTC on 24 April 2026 – We initiated a rollback to the last known good version in AZ-02. • 21:02 UTC on 24 April 2026 – We observed instability reappear in AZ-03, determined that this was because the rollback had not yet completed across all update domains, so manually reinitialized the rollback. • 22:39 UTC on 24 April 2026 – We confirmed the rollback was fully completed in AZ-03. • 23:22 UTC on 24 April 2026 – All customer impact confirmed as mitigated. How are we making incidents like this less likely or less impactful? • We assessed the risk of occurrence in other regions with large Availability Zones and rolled back the PubSub service update in these regions. (Completed) • We are improving our test coverage surrounding the failure cases and load patterns that contributed to this incident, to catch similar issues before they reach production. (Estimated completion: June 2026) • We are reducing the blast radius of each partition by adjusting the scale of the partitioning used by the PubSub service. (Estimated completion: June 2026) • We are investing in additional enrichment surrounding our Network Resource Provider monitoring, to determine impacted subscriptions and send initial communications automatically. (Estimated completion: August 2026) • While co-locating compute and data on the same node in Service Fabric provides performance benefits, it can make rollback and recovery more challenging under resource-constrained conditions. To address this, we are developing a solution designed to ensure resource constraints do not delay recovery operations. (Estimated completion: September 2026) • Finally, we are further developing our AI-assisted communication system to consolidate relevant incident information into more detailed customer updates. (Estimated completion: September 2026) How can customers make incidents like this less impactful? • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/azure/architecture/patterns/geodes and https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/5GP8-W0G

Read the full incident report →

Started Mar 09, 2026, 11:20 PM UTC· resolved Mar 10, 2026, 07:32 PM UTC

1. postmortem Mar 10, 2026, 07:32 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/8GCS-858 What happened? Between 23:20 UTC on 9 March and 19:32 UTC on 10 March 2026, a platform issue resulted in impact to the Azure OpenAI Service. Impacted customers experienced HTTP 400 and HTTP 429 error responses, specifically for the GPT-5.2 model. All other GPT models were unaffected during this time. This incident impacted customer resources and queries in the following regions: Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central, and UK South. What went wrong and why? The Azure OpenAI Service processes customer requests through model engines deployed across multiple Azure regions and supported by traffic routing systems. Depending on the selected deployment model (Global, Data Zone, or Regional), note that requests may be routed across multiple Azure regions within defined geographic boundaries to support availability and resilience, while customer data remains stored at rest in the selected Azure region. Learn more at: https://azure.microsoft.com/blog/enterprise-trust-in-azure-openai-service-strengthened-with-data-zones. A recent update to the Azure OpenAI GPT 5.2 model introduced a configuration change that was not compatible with the version of the model engine code running in production. As part of this update, certain feature settings were enabled to improve service efficiency and resilience – however, the deployed engine version did not yet support those settings. As a result, when customer requests were routed to engines with this mismatch, the service was unable to process these requests correctly. Generally speaking, any updates to the service are rolled out in line with our Safe Deployment Practices (SDP) which deploys to different regions gradually, in stages. During this update, the earlier stages of the rollout did not include sufficient backend model instances for this issue to surface before the update progressed to additional regions. As a result, the rollout had completed its deployment across our fleet before we were able to determine customer impact. During mitigation of the primary issue, we identified a secondary issue that affected service recovery. Azure OpenAI relies on internal telemetry to understand real-time service capacity across regions and to route traffic accordingly. At the time recovery actions were underway, an unrelated issue in this internal telemetry system led to incomplete capacity information being leveraged. As a result, traffic routes were temporarily being determined using incomplete data, which led to a disproportionate amount of traffic being directed to a limited set of available regions. This created additional resource pressure in those regions and resulted in continued intermittent request failures (HTTP 429 errors) for some customers, even as the rollback of the configuration issue was progressing and other regions were actually available to receive requests. Once the routing updates were successfully completed and full capacity information was restored across regions, traffic distribution normalized and service recovery progressed as expected. During the incident, we also identified a monitoring gap related to anomalous HTTP 400 error patterns. While HTTP 400 responses do occur during normal service usage, our monitoring was not configured for service‑side anomalies, only for client-side errors – which are typically caused by incorrect parameters in user requests. This miss in monitoring during the initial stages of the incident delayed detection and response. How did we respond? • 23:20 UTC on 09 March 2026 – Customer impact began, triggered by the recent service update. • 00:19 UTC on 10 March 2026 – We detected the issue via service monitoring. This prompted us to begin our investigation, engage with other teams to troubleshoot, and start developing a hot fix. • 03:18 UTC on 10 March 2026 – We determined that a rollback could mitigate more quickly than hotfix, so started the rollback for the impacted model. • 10:55 UTC on 10 March 2026 – Traffic routes were determined to be using incomplete data, due to the aforementioned dependency issue. • 12:40 UTC on 10 March 2026 – We identified and investigated resource constraints. • 18:00 UTC on 10 March 2026 – Rollback actions completed across all affected regions. • 19:30 UTC on 10 March 2026 – Full capacity information was restored across regions, traffic distribution normalized. • 19:32 UTC on 10 March 2026 – Once monitoring confirmed stable recovery, we determined the service was fully restored and all customer impact had been mitigated. How are we making incidents like this less likely or less impactful? • We have already conducted additional engineer training on our operating procedures – including which scenarios can be quickly rolled back – to reduce the time to mitigate similar issues. (Completed) • We have incorporated storing a ‘cached’ known good version of the traffic routing details, as an additional layer of resilience in case the dependent service is not able to serve the latest information on regional capacity availability. (Completed) • We have improved our monitoring surrounding HTTP 400 errors, by establishing thresholds of errors on the service side. (Completed) • To expand that monitoring further, we are improving our anomaly detection surrounding HTTP 4xx errors – to alert on anomalous error rates that may not meet our usual thresholds. (Estimated completion: April 2026) • We are incorporating additional signals to our deployment systems, to reduce potential impact by stopping problematic rollouts automatically. (Estimated completion: May 2026) • Finally, we are improving our safe deployment practices by ensuring that early stages have sufficient backend model instances to catch issues like this earlier. (Estimated completion: June 2026) How can customers make incidents like this less impactful? • Consider reviewing our guidance and best practices related to Business Continuity and Disaster Recovery (BCDR) scenarios for Azure OpenAI: https://learn.microsoft.com/azure/ai-services/openai/how-to/business-continuity-disaster-recovery • Consider reviewing and implementing our best practices surrounding retry patterns, especially with exponential backoff, to improve workload resiliency during intermittent issues: https://learn.microsoft.com/azure/architecture/patterns/retry • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/8GCS-858

Read the full incident report →

Started Feb 07, 2026, 07:58 AM UTC· resolved Feb 08, 2026, 04:24 AM UTC

1. postmortem Feb 08, 2026, 04:24 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/_SVS-5_G What happened? Between 07:58 UTC on 07 February 2026 and 04:24 UTC on 08 February 2026, impacted customers experienced intermittent service unavailability, timeouts and/or higher than normal latency for services in the West US region. The event began following a power interruption affecting one of the datacenters within the region, after which impact manifested as infrastructure availability loss and service disruptions across multiple dependent workloads in the region. As power stabilization progressed, recovery proceeded in phases – however, a subset of storage and compute infrastructure components did not return to a healthy state immediately after power restoration, which slowed recovery for dependent components, and contributed to ongoing symptoms such as delayed telemetry and resource recovery. Impacted services included: • Azure AI Search: Customers using Azure AI Search in West US may have experienced query failures, indexing delays, or temporarily unavailable search endpoints. • Azure App Service: Between 07:58 UTC and approximately 09:45 UTC on 07 February 2026, customers using Azure App Service in West US may have experienced HTTP 500 errors, request timeouts, or failed deployments. • Azure Backup: During the impact window, backup jobs in West US may have experienced delays or temporary failures due to storage unavailability. • Azure Cache for Redis: Customers may have observed connectivity failures, increased latency, or transient cache unavailability in West US. • Azure Container Registry: Customers may have experienced delays or failures when pulling or pushing container images in West US. • Azure Cosmos DB: Customers may have experienced a degradation in service availability and/or request latency. Some requests may have resulted in server errors or timeouts. • Azure Data Factory: Customers may have observed pipeline execution failures or delays in West US. Affected integration runtimes recovered following restoration of underlying networking and storage services. • Azure Database for MySQL – Flexible Server: Between 09:40 UTC and 20:50 UTC on 07 February 2026, customers may have experienced connection failures or degraded database availability in West US. • Azure Database for PostgreSQL – Flexible Server: Customers may have experienced database connectivity interruptions or degraded performance in West US. • Azure Databricks: Customers may have experienced workspace unavailability, cluster startup failures, or job execution interruptions in West US. • Azure DevOps: Customers may have experienced delays in log ingestion and analytics reporting in West US. • Azure Event Hubs: Between 07:58 UTC and 19:05 UTC on 7 February 2026, customers may have experienced failures or high latency when sending and processing events. Control plane operations like creating/updating/deleting Event Hubs or Consumer Groups were also impacted. • Azure IoT Hub: Customers may have observed delayed message processing or temporary endpoint unavailability in West US. • Azure Kubernetes Service (AKS): Between 07:58 UTC and 21:46 UTC on 07 February 2026, customers using AKS in West US may have experienced control plane instability, node health reporting delays, or pod scheduling failures. • Azure Key Vault: Customers may have experienced intermittent authentication or secret retrieval failures in West US. • Azure Monitor: Customers may have observed delayed metrics, log ingestion latency, or temporary alerting gaps in West US. • Azure Monitor (Application Insights): Between 07:58 UTC on 07 February 2026 and 04:24 UTC on 08 February 2026, customers may have experienced delayed telemetry ingestion and incomplete monitoring data in West US. • Azure Service Bus: Between 07:58 UTC and 19:05 UTC on 7 February 2026, customers may have experienced transient messaging failures, delayed message processing, or control plane disruptions impacting queues and topics in West US. • Azure SQL Database: Customers may have experienced SQL error 40613 (database unavailable) or connectivity interruptions in West US. Service mitigation included database failover and host recovery. • Azure Storage: Customers may have experienced read/write failures, increased latency, or temporary storage account unavailability in West US. Storage services were restored in phases as power and network infrastructure stabilized. • Azure Stream Analytics: Customers may be experiencing service availability disruptions, including delays or failures in data ingestion, processing, or output delivery. • Azure Virtual Machines: Customers may have experienced VM start failures, allocation errors, or temporary unavailability of running instances in West US. • Azure Virtual Machines (Confidential VM types): Customers may have experienced VM deployment or startup failures in West US, due to dependent storage disruption. • Microsoft Defender for Cloud Apps: Customers may have observed delayed alerts or telemetry ingestion in West US during the recovery period. What went wrong and why? Under normal operating conditions, Azure datacenter infrastructure is designed to tolerate utility power disturbances through redundant electrical systems and automated failover to on-site generators. During this incident, although utility power was still active, an electrical failure in an onsite transformer resulted in the loss of utility power to the datacenter. Although generators started as designed, a cascading failure within a control system prevented the automated transfer of load from utility power to generator power. As a result, Uninterruptible Power Supply (UPS) batteries carried the load for several minutes – until they were fully depleted, leading to customer impact from this power loss as early as 07:58 UTC. Once power loss occurred, recovery was not uniform across all dependent infrastructure. Our datacenter operations team restored power to IT racks by leveraging our onsite generators, with 90% of IT racks powered by 09:31 UTC on 07 February. While infrastructure generally came back online as expected after power restoration, subsets of networking, storage and compute infrastructure required further electrical control system troubleshooting, before power could be restored. We were fully restored and running generator-backed power by 11:29 UTC. Power restoration was sequenced to avoid destabilizing power that had already been brought back online, and due to the inconsistent state of the control system. Following power restoration at the datacenter, recovery of customer workloads progressed in stages based on dependencies. Network and Storage recovery were critical requirements, as compute hosts must be able to access persistent storage to complete boot, validate system state, and reattach customer data before workloads can resume. Network devices and services recovered as expected once power was restored. The power loss affected six storage scale units within the datacenter, of which four recovered as expected once power was restored. Two storage scale units experienced prolonged recovery, which became a primary factor contributing to extended impact as these contained a large subset of storage nodes that failed to complete the boot process. During the boot process, critical artifacts need to be downloaded, but these were delayed or timed out – resulting in these nodes entering a state for which there was not an automated recovery action. Manual operations were attempted, and eventually succeeded in recovering the necessary nodes to bring these storage scale units back online. Due to the dependencies that many compute and platform services have on these storage scale units, there was a delay in overall service restoration. Storage recovery completed by 19:05 UTC. Compute recovery began once power and storage dependencies were partially restored. Automated recovery systems were available and functioning, but the scale of the event resulted in a surge of required recovery activities across the datacenter. This placed sustained pressure on regional shared infrastructure components responsible for coordinating health checks, power state validation, and repair actions. During the initial recovery window, this elevated load reduced the speed at which individual compute nodes could be recovered. While many nodes returned to service as expected, some nodes required repeated recovery attempts, due to insufficient prioritization across concurrent fault categories during the surge of recovery activities. We introduced optimizations to accelerate recovery, including deprioritizing non-critical recovery activities. As pressure on these shared systems decreased, compute recovery progressed more predictably and remaining nodes were gradually restored. Compute recovery completed by 23:30 UTC. How did we respond? • 07:54 UTC on 07 February 2026 – Initial electrical failure in an onsite transformer, although various UPS batteries carried the load for several minutes. • 07:58 UTC on 07 February 2026 – Customers began experiencing unavailability and delayed monitoring/log data. • 08:07 UTC on 07 February 2026 – Datacenter and engineering teams engaged and initiated coordinated investigation across power, network, storage, and compute recovery workstreams. • 09:31 UTC on 07 February 2026 – 90% of facility power restored on generators. • 11:26 UTC on 07 February 2026 – 93% of facility power restored on generators. • 11:29 UTC on 07 February 2026 – 100% of facility power restored on generators with continued recovery of hosted services. • 12:15 UTC on 07 February 2026 – Storage recovery started and dependent services started seeing recovery. • 15:00 UTC on 07 February 2026 – Targeted remediation actions continued for remaining unhealthy services. • 15:25 UTC on 07 February 2026 – Applied configuration changes to optimize processing of recovery events, to assist in expediting the recovery of compute services. • 19:05 UTC on 07 February 2026 – Storage recovery was complete. • 19:24 UTC on 07 February 2026 – Compute recovery system stabilized and fully operational, continued gradually restoring remaining nodes. • 23:30 UTC on 07 February 2026 – Long tail of Compute nodes impact mitigated. • 04:24 UTC on 08 February 2026 – All affected services confirmed mitigated. • 03:42 UTC on 09 February 2026 – Power was transitioned back to utility power, restoring the datacenter to normal operations. How are we making incidents like this less likely or less impactful? • The control system and dependent components have been inspected, reconfigured, and validated to support transfer-to-generator events more reliably. (Completed) • We have reevaluated how best to prioritize critical infrastructure earlier in the power restoration sequence, to ensure a smoother recovery across dependent services. (Completed) • We are working to enhance the recovery system through prioritization and throttling, to improve stability during surge scenarios like this one. (Estimated completion: July 2026) • Finally, we are continually refining how our infrastructure and services come back online after datacenter power loss events, to make these more robust to different failure scenarios. How can customers make incidents like this less impactful? • Since Availability Zones are not available in the West US region, customers should consider a multi-region geodiversity strategy for mission-critical workloads: https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones#deployment-approach-4-multi-region-deployments and https://learn.microsoft.com/azure/well-architected/design-guides/disaster-recovery • Consider leveraging data redundancy options such as Read-Access Geo-Redundant Storage (RA-GRS) to ensure data accessibility during regional incidents: https://learn.microsoft.com/azure/storage/common/storage-redundancy • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/_SVS-5_G

Read the full incident report →

Started Feb 02, 2026, 06:03 PM UTC· resolved Feb 03, 2026, 12:30 AM UTC

1. postmortem Feb 03, 2026, 12:30 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/FNJ8-VQZ What happened? Between 18:03 UTC on 02 February 2026 and approximately 00:30 UTC on 03 February 2026, a platform issue caused some customers to experience degraded performance and control plane failures, for multiple Azure services across multiple regions. Impacted services included: • Azure Virtual Machines (VMs) – Customers may have experienced failures when deploying or scaling virtual machines, including errors during provisioning and lifecycle operations. • Azure Virtual Machine Scale Sets (VMSS) – Customers may have experienced failures when scaling instances or applying configuration changes. • Azure Kubernetes Service (AKS) – Customers may have experienced failures in node provisioning and extension installation. • Azure DevOps (ADO) and GitHub Actions – Customers may have experienced pipeline failures when tasks required VM extensions or related packages. Actions jobs queued and timed out while waiting to acquire a hosted runner. Other GitHub features that leverage this compute infrastructure were similarly impacted, including Copilot Coding Agent, Copilot Code Review, CodeQL, Dependabot, GitHub Enterprise Importer, and Pages. All regions and runner types were impacted. • Other dependent services – Customers may have experienced degraded performance or failures in operations that required downloading extension packages from Microsoft-managed storage accounts, including Azure Arc enabled servers, and Azure Database for PostgreSQL. As part of our recovery, between 00:08 UTC and 06:05 UTC on 03 February 2026, customers using the 'Managed identities for Azure resources' service (formerly Managed Service Identity) may have experienced failures when attempting to create, update, or delete Azure resources, or acquire Managed Identity tokens – in the East US and West US regions. Originally communicated on incident tracking ID _M5B-9RZ, this impacted the creation and management of Azure resources with assigned managed identities – including but not limited to Azure AI Video Indexer, Azure Chaos Studio, Azure Container Apps, Azure Database for PostgreSQL Flexible Servers, Azure Databricks, Azure Kubernetes Service, Azure Stream Analytics, Azure Synapse Analytics, and Microsoft Copilot Studio. Newly created resources of these types may have also failed to authenticate using managed identity. What went wrong, and why? On 02 February 2026, a remediation workflow executed based on a resource policy, intended to disable anonymous access on Microsoft-managed storage accounts. The purpose of this policy is to reduce unintended exposure by ensuring that anonymous access is not enabled unless explicitly required. Due to a data synchronization problem in the targeting logic, the policy was incorrectly applied to a subset of storage accounts that are intentionally configured to allow anonymous read access for platform functionality. These storage accounts form part of the VM extension package storage layer. VM extensions are small applications used during provisioning and lifecycle operations to configure, secure, and manage VMs. Extension artifacts are retrieved directly from storage accounts, during early-stage provisioning and scaling workflows. When anonymous access was disabled across these storage accounts, VMs and dependent services were unable to retrieve required extension artifacts. This resulted in control plane failures and degraded performance across affected services. Although the policy process followed safe deployment practices and was initially rolled out region by region, the health assessment defect caused the incorrect targeting state to propagate broadly across public cloud regions in a short time window. This accelerated the impact beyond the intended blast radius. Once identified, the policy job was immediately disabled to prevent further changes. Rollback procedures to restore anonymous access were validated in a test region and then executed region by region across affected storage accounts. As access was restored, control plane operations gradually recovered and service management queues began to clear. As storage access was restored, a secondary issue emerged. The re-enablement of the VM extension package storage layer allowed multiple previously blocked workflows to resume concurrently, across infrastructure orchestration systems and dependent services. The resulting surge in resource management operations generated a significant and sudden increase in requests to the backend service supporting Managed Identities for Azure resources. As a result, starting at 00:08 UTC on 03 February 2026, customers in East US and West US experienced failures when attempting to create, update, or delete Azure resources that use managed identities, as well as when acquiring Managed Identity tokens. Automatic retry behaviors in upstream components, designed to provide resiliency under transient failure conditions, amplified traffic against the managed identity backend. The cumulative effect exceeded service limits in East US. Resilient routing mechanisms subsequently directed retry traffic to West US, resulting in similar saturation. Although the managed identity infrastructure was scaled out, additional capacity alone did not immediately mitigate impact – because retry amplification continued to generate load at a rate that exceeded recovery capacity. Active load shedding and controlled reintroduction of traffic were required to stabilize the service and allow backlog processing while maintaining normal operations, with full recovery by 06:05 UTC. How did we respond? • 18:03 UTC on 02 February 2026 – Customer impact began, triggered by the periodic remediation workflow starting. • 18:29 UTC on 02 February 2026 – Internal service monitoring detected a subset of regions having an increasing number of control plane failures. • 19:46 UTC on 02 February 2026 – Correlated issues across multiple regions. • 19:55 UTC on 02 February 2026 – Service monitoring detected failure rates exceeding failure limit thresholds. • 20:10 UTC on 02 February 2026 – We began collaborating to devise a mitigation solution and investigate the underlying factors. • 21:15 UTC on 02 February 2026 – We applied a primary proposed mitigation and validated that it was successful on a test instance. • 21:18 UTC on 02 February 2026 – We identified and disabled the remediation workflow and stopped any ongoing activity, so it would not impact additional storage accounts. • 21:50 UTC on 02 February 2026 – Began broader mitigation to impacted storage accounts. Customers saw improvements as work progressed. • 00:07 UTC on 03 February 2026 – Storage accounts for a few high volume VM extensions that utilize managed identity finished their re-enablement in East US. • 00:08 UTC on 03 February 2026 – Unexpectedly, customer impact increased first in East US and cascaded to West US with retries, as a critical managed identity service degraded under recovery load. • 00:14 UTC on 03 February 2026 – Automated alerting identified availability impact to Managed Identity services in East US. Engineers quickly recognized that the service was overloaded and began to scale out. • 00:30 UTC on 03 February 2026 – All extension hosting storage accounts had been reenabled, mitigating this impact in all regions other than East US and West US. • 00:50 UTC on 03 February 2026 – Initial scale outs of managed identity service infrastructure scale out completed, but the new resources were still unable to handle the traffic volume due to the increasing backlog of retried requests. • 02:00 UTC on 03 February 2026 – A second, larger set of managed identity service scale outs completed. Once again, the capacity was unable to handle the volume of backlogs and retries. • 02:15 UTC on 03 February 2026 – Reviewed additional data and monitored downstream services to ensure that all mitigations were in place for all impacted storage accounts. • 03:55 UTC on 03 February 2026 – To recover infrastructure capacity for the managed identity service, we began rolling out a change to remove all traffic so that the infrastructure could be repaired without load. • 04:25 UTC on 03 February 2026 – After infrastructure nodes recovered, we began gradually ramping traffic to them, allowing backlogged identity operations to begin to process safely. • 06:05 UTC on 03 February 2026 – Backlogged operations completed, and services returned to normal operating levels. We concluded our monitoring and confirmed that all customer impact had been mitigated. How are we making incidents like this less likely or less impactful? • We are strengthening traffic controls and throttling so that unexpected load spikes or retry storms are contained early, and cannot overwhelm the managed identity service or impact other customers. (Estimated completion: February 2026) • We are working to fix the data synchronization problem, by addressing detection gaps and code defects with validation coverage, to protect from similar scenarios. (Estimated completion: March 2026) • We will exercise and optimize the rollback of the remediation workflow, including automation where appropriate, to mitigate issues more quickly. (Estimated completion: April 2026) • We are increasing capacity headroom and tightening scaling safeguards, to ensure sufficient resources remain available during recovery events and sudden demand. (Estimated completion: April 2026) • We will improve orchestration of the remediation workflow to be more in line with our Safe Deployment Practices, integrated with service health indicators, to reduce multi-region failure risk. (Estimated completion: May 2026) • We are improving retry behavior and internal efficiency, so the service degrades more gracefully under stress. (Estimated completion: June 2026) • We are improving regional isolation and failover behavior, to prevent issues in one region from cascading into paired regions during failures or recovery. (Estimated completion: July 2026) How can customers make incidents like this less impactful? • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. • Note that the impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/FNJ8-VQZ

Read the full incident report →

Started Dec 22, 2025, 08:05 AM UTC· resolved Dec 22, 2025, 06:30 PM UTC

1. postmortem Dec 22, 2025, 06:30 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/FV31-PQG What happened? Between 08:05 UTC and 18:30 UTC on 22 December 2025, the Microsoft Entra Privileged Identity Management (PIM) service experienced issues that impacted role activations for a subset of customers. Some requests returned server errors (5xx) and, for a portion of customers, some requests returned unexpected client errors (4xx). This issue manifested as API failures, elevated latency, and various activation errors. Impacted actions could have included: • Reading eligible or active role assignments • Eligible or Active Role assignment management • PIM Policy and PIM Alerts management • Activation of Eligible or Active Role assignments • Deactivation of role assignments initiated by customer (deactivations triggered by the expiration of previous activation were not impacted) • Approval of role assignment activation or extension Customer initiated PIM operations from various Microsoft management portals, mobile app, and API calls were likely impacted. During this incident period, operations that were retried may have succeeded. What went wrong and why? Under normal conditions, Privileged Identity Management processes requests—such as role activations—by coordinating activity across its front-end APIs, traffic routing layer, and the backend databases that store role activation information. These components work together to route requests to healthy endpoints, complete required checks, and process activations without delay. To support this flow, the system maintains a pool of active connections for responsiveness and automatically retries brief interruptions to keep error rates low, helping ensure customers experience reliable access when reading, managing, or activating role assignments. As part of ongoing work to improve how the PIM service stores and manages data, configuration changes which manage the backend databases were deployed incrementally using safe deployment practices. The rollout progressed successfully through multiple rings, with no detected errors and all monitored signals remaining within healthy thresholds. When the deployment reached a ring operating under higher workload, the additional per-request processing increased demand on the underlying infrastructure that hosts the API which manages connections to the backend databases. Although the service includes throttling mechanisms designed to protect against spikes in API traffic, this scenario led to elevated CPU utilization without exceeding request-count thresholds, so throttling did not engage. Over time, the sustained load caused available database connections to be exhausted, and the service became unable to process new requests efficiently. This resulted in delays, timeouts, and errors for customers attempting to view or activate privileged roles. How did we respond? • 21:36 UTC on 15 December 2025 – The configuration change deployment was initiated. • 22:08 UTC on 19 December 2025 – The configuration change was progressed to the ring with the heavy workload. • 08:05 UTC on 22 December 2025 – Initial customer impact began. • 08:26 UTC on 22 December 2025 – Automated alerts received for intermittent, low volume of errors prompting us to start investigating. • 10:30 UTC on 22 December 2025 – We attempted isolated restarts on impacted database instances in an effort to mitigate low-level impact. • 13:03 UTC on 22 December 2025 – Automated monitoring alerted us to elevated error rates and a full incident response was initiated. • 13:22 UTC on 22 December 2025 – We identified that calls to the database were intermittently timing out. Traffic volume appeared to be normal with no significant surge detected however we observed the spike in CPU utilization. • 13:54 UTC on 22 December 2025 – Mitigation efforts began, including beginning to scale out the impacted environments. • 15:05 UTC on 22 December 2025 – Scale out efforts were observed as decreasing error rates but not completely eliminating failures. Further instance restarts provided temporary relief. • 15:25 UTC on 22 December 2025 – Scaling efforts continued. We engaged our database engineering team to help investigate. • 16:37 UTC on 22 December 2025 – While we did not correlate the deployment to this incident, we initiated a rollback of the configuration change. • 17:20 UTC on 22 December 2025 – Scale-out efforts completed. • 17:45 UTC on 22 December 2025 – Service availability telemetry was showing improvements. Some customers began to report recovery. • 18:30 UTC on 22 December 2025 – Customer impact confirmed as mitigated, after rollback of configuration change had completed and error rates had returned to normal levels. How are we making incidents like this less likely or less impactful? • We have rolled back the problematic configuration change across all regions. (Completed) • For outages that manifest later as a result of configuration updates, we are developing a mechanism to help engineers correlate these signals more quickly. (Estimated completion: January 2026) • We are working to ensure this configuration change will not inadvertently introduce excessive load before we redeploy this again. (Estimated completion: January 2026) • We are working on updating our auto-scale configuration to be more responsive to changes in CPU usage. (Estimated completion: January 2026) • We are enabling monitoring and runbooks for available database connections to respond to emerging issues sooner. (Estimated completion: February 2026) How can customers make incidents like this less impactful? • Consider retrying critical operations directly through the API, for example, when requesting just in time access: https://learn.microsoft.com/entra/id-governance/privileged-identity-management/pim-apis#iteration-3-current--pim-for-microsoft-entra-roles-groups-in-microsoft-graph-api-and-for-azure-resources-in-azure-resource-manager-api • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/FV31-PQG

Read the full incident report →

Started Dec 08, 2025, 12:00 AM UTC· resolved Dec 08, 2025, 12:00 AM UTC

1. postmortem Dec 08, 2025, 12:00 AM UTC

   What happened? Between 11:04 and 14:13 EST on 08 December 2025, customers using any of the Azure Government regions may have experienced failures when attempting to perform service management operations through Azure Resource Manager (ARM). This included operations attempted through the Azure Portal, Azure REST APIs, Azure PowerShell, and Azure CLI. Affected services included but were not limited to: Azure App Service, Azure Backup, Azure Communication Services, Azure Data Factory, Azure Databricks, Azure Functions, Azure Kubernetes Service, Azure Maps, Azure Migrate, Azure NetApp Files, Azure OpenAI Service, Azure Policy (including Machine Configuration), Azure Resource Manager, Azure Search, Azure Service Bus, Azure Site Recovery, Azure Storage, Azure Virtual Desktop, Microsoft Fabric, and Microsoft Power Platform (including AI Builder and Power Automate). What went wrong and why? Azure Resource Manager (ARM) is the gateway for management operations for Azure services. ARM does authorization for these operations based on authorization policies, stored in Cosmos DB accounts that are replicated to all regions. On 08 December 2025, an inadvertent automated key rotation resulted in ARM failures to fetch authorization policies that are needed to evaluate access. As a result, ARM was temporarily unable to communicate with underlying storage resources, causing failures in service-to-service communication and affecting resource management workflows across multiple Azure services. This issue surfaced as authentication failures and 500 Internal Server errors to customers across all clients. Because the content of the Cosmos DB accounts for authorization policies is replicated globally, all regions within the Azure Government cloud were affected. Microsoft services use an internal system to manage keys and secrets, which also makes it easy to perform regular needed maintenance activities, such as rotating secrets. Protecting identities and secrets is a key pillar in our Secure Future Initiative to reduce risk, enhance operational maturity, and proactively prepare for emerging threats to identity infrastructure – by prioritizing secure authentication and robust key management. In this case, our ARM service was using a key in a ‘manual mode’ which means that any key rotations would need to be manually coordinated, so that traffic could be moved to use a different key before the key could be regenerated. The Cosmos DBs that ARM use for accessing authorization policies was intentionally onboarded to the Microsoft internal service which governs the account key lifecycle, but unintentionally configured with the option to automatically rotate the keys enabled. This automated rotation should have been disabled as part of the onboarding process, until such time as it was ready to be fully automated. Earlier on the same day of this incident, a similar key rotation issue affected services in the Azure in China sovereign cloud. Both the Azure Government and Azure in China sovereign cloud had their (separate but equivalent) keys created on the same day, starting completely independent timers, back in February 2025 – so each was inadvertently rotated on their respective timers, approximately three hours apart. As such, the key used by ARM for the Azure Government regions was automatically rotated, before the same key rotation issue affecting the Azure in China regions was fully mitigated. Although potential impact to other sovereign clouds was discussed as part of the initial investigation, we did not have a sufficient understanding of the inadvertent key rotation to be able to prevent impact in the second sovereign cloud, Azure Government. How did we respond? • 11:04 EST on 08 December 2025 – Customer impact began. • 11:07 EST on 08 December 2025 – Engineering was engaged to investigate based on automated alerts. • 11:38 EST on 08 December 2025 – We began applying a fix for the impacted authentication components. • 13:58 EST on 08 December 2025 – We began to restart ARM instances, to speed up the mitigation process. • 14:13 EST on 08 December 2025 – All customer impact confirmed as mitigated. How are we making incidents like this less likely or less impactful? • First and foremost, our ARM team have conducted an audit to ensure that there are no other manual keys that are misconfigured to be auto-rotated, across all clouds. (Completed) • Our internal secret management system has paused automated key rotations for managed keys, until usage signals are made available on key usage – see the Cosmos DB change safety repair item below. (Completed) • We will complete the migration to auto-rotated Cosmos DB account keys for ARM authentication accounts, across all clouds. (Estimated completion: February 2026) • Our Cosmos DB team will introduce change safety controls that block regenerating keys that have usage, by emitting a relevant usage signal. (Estimated completion: Public Preview by April 2026, General Availability to follow by August 2026) How can customers make incidents like this less impactful? • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/ML7_-DWG

Read the full incident report →

Started Dec 08, 2025, 12:00 AM UTC· resolved Dec 08, 2025, 12:00 AM UTC

1. postmortem Dec 08, 2025, 12:00 AM UTC

   What happened? Between 16:50 CST on 08 December 2025 and 02:00 CST on 09 December 2025 (China Standard Time) customers using any of the Azure in China regions may have experienced failures when attempting to perform service management operations through Azure Resource Manager (ARM). This included operations attempted through the Azure Portal, Azure REST APIs, Azure PowerShell, and Azure CLI. Affected services included but were not limited to: Azure AI Search, Azure API Management, Azure App Service, Azure Application Insights, Azure Arc, Azure Automation, Azure Backup, Azure Data Factory, Azure Databricks, Azure Database for PostgreSQL Flexible Server, Azure Kubernetes Service, Azure Logic Apps, Azure Managed HSM, Azure Marketplace, Azure Monitor, Azure Policy (including Machine Configuration), Azure Portal, Azure Resource Manager, Azure Site Recovery, Azure Stack HCI, Azure Stream Analytics, Azure Synapse Analytics, and Microsoft Sentinel. What went wrong and why? Azure Resource Manager (ARM) is the gateway for management operations for Azure services. ARM does authorization for these operations based on authorization policies, stored in Cosmos DB accounts that are replicated to all regions. On 08 December 2025, an inadvertent automated key rotation resulted in ARM failures to fetch authorization policies that are needed to evaluate access. As a result, ARM was temporarily unable to communicate with underlying storage resources, causing failures in service-to-service communication and affecting resource management workflows across multiple Azure services. This issue surfaced as authentication failures and 500 Internal Server errors to customers across all clients. Because the content of the Cosmos DB accounts for authorization policies is replicated globally, all regions within the Azure in China cloud were affected. Azure services use an internal system to manage keys and secrets, which also makes it easy to perform regular needed maintenance activities, such as rotating secrets. Protecting identities and secrets is a key pillar in our Secure Future Initiative to reduce risk, enhance operational maturity, and proactively prepare for emerging threats to identity infrastructure – by prioritizing secure authentication and robust key management. In this case, our ARM service was using a key in a ‘manual mode’ which means that any key rotations would need to be manually coordinated, so that traffic could be moved to use a different key before the key could be regenerated. The Cosmos DBs that ARM use for accessing authorization policies was intentionally onboarded to the internal service which governs the account key lifecycle, but unintentionally configured with the option to automatically rotate the keys enabled. This automated rotation should have been disabled as part of the onboarding process, until such time as it was ready to be fully automated. How did we respond? • 16:50 CST on 08 December 2025 – Customer impact began. • 16:59 CST on 08 December 2025 – Engineering was engaged to investigate based on automated alerts. • 18:37 CST on 08 December 2025 – We identified the underlying cause as the incorrect key rotation. • 19:16 CST on 08 December 2025 – We identified mitigation steps and began applying a fix for the impacted authentication components. This was tested and validated before being applied. • 22:00 CST on 08 December 2025 – We began to restart ARM instances, to speed up the mitigation process. • 23:53 CST on 08 December 2025 – Many services had recovered but residual impact remained for some services. • 02:00 CST on 09 December 2025 – All customer impact confirmed as mitigated. How are we making incidents like this less likely or less impactful? • First and foremost, our ARM team have conducted an audit to ensure that there are no other manual keys that are misconfigured to be auto-rotated, across all clouds. (Completed) • Our internal secret management system has paused automated key rotations for managed keys, until usage signals are made available on key usage – see the Cosmos DB change safety repair item below. (Completed) • We will complete the migration to auto-rotated Cosmos DB account keys for ARM authentication accounts, across all clouds. (Estimated completion: February 2026) • Our Cosmos DB team will introduce change safety controls that block regenerating keys that have usage, by emitting a relevant usage signal. (Estimated completion: Public Preview by April 2026, General Availability to follow by August 2026) How can customers make incidents like this less impactful? • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/JSNV-FBZ

Read the full incident report →

Started Nov 05, 2025, 04:53 PM UTC· resolved Nov 06, 2025, 02:25 AM UTC

1. postmortem Nov 06, 2025, 02:25 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/2LGD-9VG What happened? Between approximately 16:53 UTC on 5 November 2025 and 02:25 UTC on 6 November 2025, a subset of customers in the West Europe region experienced service disruptions or degraded performance across multiple services, including Virtual Machines (VM), Azure Database for PostgreSQL Flexible Server, MySQL Flexible Server, Azure Kubernetes Service, Storage, Service Bus, and Virtual Machine Scale Sets, among others. Customers using Azure Databricks in West Europe observed degraded performance when launching or scaling all-purpose and job compute workloads, which impacted Unity Catalog and Databricks SQL operations. What went wrong and why? The incident was triggered by a voltage sag in the utility grid, which caused cooling units to shut down and temperatures to rise above normal thresholds in a datacenter located in Physical Availability Zone 01 in the West Europe region. As temperature rose, multiple storage scale units automatically powered down to prevent hardware damage. Under normal circumstances, cooling systems are designed and commissioned to automatically restart after power events and are built with sufficient redundancy to maintain safe operating conditions even when failures occur. While cooling unit auto-restart capability had been regularly tested and exercised in this datacenter, the specific size and duration of utility sag exposed a previously unknown hardware issue that prevented the cooling units from restarting. This resulted in temperatures exceeding safe operational limits for infrastructure. The extended recovery time was influenced by several factors. Recovery of storage scale units was prolonged because some storage servers entered a degraded state, necessitating sequential validation before they could be brought back online. In addition, dependent services such as compute and networking could not resume until storage integrity checks were complete. Finally, additional thermal audits were performed before workloads were reactivated to ensure that no residual risk remained. How did we respond? Automated monitoring detected the temperature anomaly and triggered an incident response. Facilities teams restored safe operating conditions by performing hard reset on the affected units and confirmed that temperatures were trending back to safe levels. Service recovery efforts began once the facility had reached safe operating temperatures. Following this, storage systems initiated comprehensive data consistency checks when nodes in a storage scale unit restarted after the unplanned shutdown. These checks, which may include reconstructing replicas that have fallen behind. The recovery process prioritizes data integrity and durability over availability, resulting in customer traffic being blocked until all integrity validations was completed and contributing to the extended recovery time. In parallel, engineers prioritized recovery of compute hosts to stop further VM failures and allow dependent services to resume. Timeline of events: • 16:53 UTC on 5 November 2025 – Customer impact began as elevated temperatures caused multiple storage scale units to shut down. • 16:55 UTC on 5 November 2025 – Datacenter monitoring detected temperature breaches and triggered thermal alerts. • 17:20 UTC on 5 November 2025 – Power sag and cooling unit failure to restart identified as contributing factors. Cooling recovery begins as engineers manually restart cooling units. • 17:40 UTC on 5 November 2025 – Mitigation workstream started as cooling restoration progresses. • 17:48 UTC on 5 November 2025 – All cooling units restored, we start seeing temperatures reducing as cooling normalized. • 17:50 UTC on 5 November 2025 – Rack-level thermal monitoring returned to safe operational thresholds. Engineering teams initiated storage recovery. • 18:30 UTC on 5 November 2025 – Sequential storage scale unit validation began. Each scale unit underwent integrity checks to ensure no data corruption before bringing storage services back into rotation. • 20:00 UTC on 5 November 2025 – Gradual restoration of storage scale units. Recovery was staged to avoid overloading power and cooling systems. Dependent compute nodes remained offline during this time. • 23:30 UTC on 5 November 2025 – Service dependency checks and orchestration. Networking and compute services were progressively re-enabled after storage scale units passed health checks. • 02:25 UTC on 6 November 2025 – All services were brought back online, and customer impact was mitigated. How are we making incidents like this less likely or less impactful? We are implementing several improvements to strengthen resilience and accelerate recovery: • We are conducting internal and manufacturer-led forensic investigations of specific device hardware and firmware to develop solutions to bolster component tolerance to power sags and voltage fluctuations. (Estimated completion: December 2025) • We are performing a gap analysis of the control circuit to ensure all relays delivering critical function of the units are powered through Uninterruptible Power Supply (UPS) and stored energy sources. (Estimated completion: January 2026) • We are validating our maintenance procedures and adding additional prechecks for Heating, Ventilation, and Air Conditioning (HVAC) subsystems. (Estimated completion: January 2026) • We are optimizing recovery steps for compute and storage systems to reduce overall restoration time. (Estimated completion: January 2026) • We are improving automated correlation between environmental signals and service health to trigger earlier, and more targeted escalations. (Estimated completion: January 2026) • We are updating our cooling unit commissioning playbooks to expand range of power sag and swell scenarios. (Estimated completion: February 2026) • In the longer term, we are implementing a new approach to accelerate service restoration after major incidents. Historically, recovery prioritized full data integrity checks before bringing services online, which extended downtime because it was a serial process. Going forward, integrity checks will run in the background during VM and disk mount operations, allowing availability to return much sooner without compromising data safety. This change is a direct outcome of this incident and represents a significant improvement in recovery speed. (Estimated completion: April 2026) How can customers make incidents like this less impactful? • Consider using Availability Zones (AZs) to run your services across physically separate locations within an Azure region. To help services be more resilient to datacenter-level failures like this one, each AZ provides independent power, networking, and cooling. Many Azure services support zonal, zone-redundant, and/or always-available configurations: https://docs.microsoft.com/azure/availability-zones/az-overview • Plan for regional redundancy by using active/active or failover designs across paired regions. • Maintain regular backups and test failover procedures to ensure recovery readiness. • Increase resilience by using retry logic and backoff strategies to prevent overload during recovery. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/2LGD-9VG

Read the full incident report →