Azure DevOps Outage History

Started Sep 09, 2026, 08:47 PM UTC· resolved Sep 10, 2026, 01:16 AM UTC

1. identified Sep 09, 2026, 08:47 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Sep 09, 2026, 10:50 PM UTC

   Our engineers are working to understand and mitigate the issue.

3. resolved Sep 10, 2026, 01:17 AM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Sep 07, 2026, 11:00 AM UTC· resolved Sep 07, 2026, 07:00 PM UTC

1. identified Sep 07, 2026, 02:14 PM UTC

   Customers may experience intermittent provisioning issues with pipelines that use MDP pools, resulting in pipeline timeouts or failures. The issue affects only a subset of MDP pool traffic and does not affect Microsoft-hosted agents. We're investigating and will provide updates as they become available.

2. resolved Sep 08, 2026, 07:08 AM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Sep 04, 2026, 07:16 AM UTC· resolved Sep 04, 2026, 12:00 PM UTC

1. identified Sep 04, 2026, 07:17 AM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Sep 04, 2026, 08:31 AM UTC

   Some npm package installation operations may fail when `npm audit` runs. Our engineers are investigating and working to restore full service availability. As a temporary workaround, run `npm install` with the `--no-audit` option: `npm install --no-audit`

3. resolved Sep 04, 2026, 12:54 PM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Sep 02, 2026, 10:01 PM UTC· resolved Sep 02, 2026, 11:26 PM UTC

1. identified Sep 02, 2026, 10:02 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Sep 02, 2026, 10:32 PM UTC

   Our engineers are currently investigating an event impacting packaging service of Azure DevOps. This event is being triaged.

3. resolved Sep 02, 2026, 11:27 PM UTC

   The issue is now fully mitigated. We throttled the identity which is causing the impact to others. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Sep 02, 2026, 06:26 AM UTC· resolved Sep 02, 2026, 06:47 AM UTC

1. identified Sep 02, 2026, 06:28 AM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. resolved Sep 02, 2026, 09:30 AM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 31, 2026, 03:50 PM UTC· resolved Aug 31, 2026, 06:45 PM UTC

1. identified Aug 31, 2026, 04:09 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Aug 31, 2026, 05:50 PM UTC

   Our engineers are working to understand and mitigate the issue.

3. identified Aug 31, 2026, 06:23 PM UTC

   We have identified the change in configuration which triggered this incident. We are reverting the change and observing impact. Will post more updates on status, root-cause and learnings/repairs from this incident.

4. identified Aug 31, 2026, 07:22 PM UTC

   We have reverted the change and services are recovering. We will continue to observe the impact and post more updates on status.

5. identified Aug 31, 2026, 07:56 PM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

6. identified Sep 02, 2026, 07:29 PM UTC

   On August 31, 2026, beginning at 16:07 UTC, customers in multiple regions experienced intermittent failures and elevated latency across Azure DevOps services, including Boards, Repos, Pipelines, Test Plans, and Artifacts. The incident was triggered by the rollout of Azure Front Door Web Application Firewall (WAF) rules designed to enhance the security posture of Azure DevOps. An issue with the configuration caused legitimate traffic to be incorrectly blocked, leading to service degradation and failures across multiple services. During the incident, approximately 0.7% of Azure DevOps commands failed and approximately 0.1% of commands experienced elevated latency. The issue was mitigated by reverting the change and restoring the last known good configuration, and service was fully restored by 19:56 UTC. We are completing a full postmortem and will use its findings to strengthen our SDP and change-rollout review processes, including validation, monitoring, and rollback safeguards to reduce the risk of recurrence. The full postmortem and associated repair items will be published once complete.

7. resolved Sep 24, 2026, 05:37 AM UTC

   On August 31, 2026, between 15:50 and 18:45 UTC, a subset of Azure DevOps customers globally experienced intermittent failures and elevated latency across core workflows including Boards, Repos, Pipelines, Test Plans, Artifacts, Analytics, and Authentication. We understand how disruptive outages can be, especially when they affect multiple services at once, and sincerely apologize to all impacted customers. ## Background Azure DevOps uses Azure Front Door (AFD) Web Application Firewall (WAF) to help protect the service from malicious traffic. WAF rules can run in audit mode, where matching requests are logged, or prevention mode, where matching requests are blocked. Before enforcing new rules broadly, we rely on recent representative audit data, service telemetry, deployment validation, and review gates to confirm that rules do not incorrectly match and block legitimate traffic. As part of ongoing security hardening, a curated subset of Microsoft Default Rule Set 2.0 rules was being enabled in prevention mode across our services by updating WAF configurations for affected Azure Front Door profiles. ## What triggered the outage? In the month of August, we began rolling out WAF configuration changes to all services, with each change taking 30 minutes to reach all AFD edge nodes globally. The WAF configuration change was triggered on the last batch of services between 15:21 and 15:48 UTC. At 15:50 UTC, a class of legitimate customer requests unexpectedly started being blocked with HTTP 403 responses. At 16:07 UTC our monitoring system started alerting us of customer impact and our engineering team quickly engaged and started investigating. At 16:17 UTC, additional legitimate requests began failing in an internal Azure DevOps service-to-service path. HTTP 403 responses returned with connection-close behavior, and due to the high volume of internal calls, repeated connection closures temporarily exhausted Source Network Address Translation (SNAT) ports in some of the Software Load Balancer (SLB) instances used by the calling service. When those ports were exhausted, the calling service could not reliably open outbound connections to dependent services, which surfaced to customers as HTTP 500 and 503 responses across a broader set of endpoints. Once the engineering team identified the WAF configuration changes as the very likely cause, we immediately started to roll them back. The first rollback configuration was created at 17:17 UTC and reduced part of the direct WAF-blocking impact. It did not fully recover the service because we had not yet identified that one internal service-to-service path was driving SNAT port exhaustion and amplifying failures across additional endpoints. We rolled back that service’s WAF configuration at 18:12 UTC, and recovery completed as the rollback propagated through AFD edge nodes by 18:45 UTC. After investigation, we discovered that the representative audit data used for validation did not include the period when the relevant rules were running in audit mode. ## What contributed to the mitigation time? Several factors increased the time required to identify and mitigate the issue: - The rollout was regionally global and applied to shared Azure Front Door configurations, so a single WAF configuration change could affect customers in multiple regions at once. - The initial false-positive block presented as both direct HTTP 403 failures and downstream symptoms across Boards, Repos, Pipelines, Test Plans, Artifacts, Analytics, and Authentication, which made it harder to separate service-local failures from the traffic-filtering issue. - Operational signals were fragmented. Early service-specific alerts and support cases did not immediately correlate with the WAF rollout, and near real-time WAF logs and monitoring for first-party traffic were insufficient to quickly identify the cause. ## Next Steps - We have reverted the problematic rule deployment and will not expand our WAF rule set in prevention mode until we harden our validation to demonstrate they do not block legitimate Azure DevOps traffic. - We are strengthening pre-deployment validation for WAF changes by requiring recent, representative audit-mode analysis, explicit architecture and security review, and guardrails that block enforcement when coverage is incomplete or stale. - We are improving telemetry and monitoring so WAF behavior, customer-impact alerts, service health signals, and support-case patterns can be correlated more quickly with Azure DevOps service health. - We are adopting staged rollouts for global and shared service resources, telemetry-based progression gates, and longer validation time between rollout stages. - We are improving rollback readiness so critical global configuration changes have pre-approved, tested rollback paths that do not depend on the failing component and reduce decision time during an incident. - We are improving our ability to troubleshoot SNAT port exhaustion and making changes to greatly reduce the likelihood of getting into a SNAT port exhaustion situation. - We are using this incident to improve how we validate, stage, monitor, and roll back global traffic-filtering changes. Sincerely, Mauro Ottaviani, Principal Group SWE Manager, Azure DevOps Engineering

Read the full incident report →

Started Aug 27, 2026, 01:30 PM UTC· resolved Aug 27, 2026, 02:10 PM UTC

1. identified Aug 27, 2026, 01:57 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. resolved Aug 27, 2026, 02:24 PM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 26, 2026, 05:55 PM UTC· resolved Aug 26, 2026, 06:20 PM UTC

1. identified Aug 26, 2026, 05:55 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. resolved Aug 26, 2026, 06:33 PM UTC

   The issue is now fully mitigated. During the outage users with orgs in Canada were unable use search functionality. This appears to be related to the earlier outage that happened in ADO. We have scaled up our compute and are now seeing availability return. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 26, 2026, 07:20 AM UTC· resolved Aug 26, 2026, 01:45 PM UTC

1. identified Aug 26, 2026, 07:20 AM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Aug 26, 2026, 08:47 AM UTC

   Our engineers are working on the mitigation of the issue, unfortunately we can not provide timeline of the mitigation yet.

3. identified Aug 26, 2026, 10:00 AM UTC

   We’re investigating an availability degradation affecting Azure DevOps in Europe. Some customers may experience intermittent delays or degraded performance across Boards, Repos, Pipelines, and Test Plans. Our engineers are working to mitigate the issue. At this time, we cannot provide an estimated time to mitigation

4. identified Aug 26, 2026, 11:07 AM UTC

   Our investigation into the Azure DevOps availability degradation in Europe is ongoing. Engineers are continuing mitigation work while they work to better understand the issue. We do not yet have a mitigation timeline and will provide another update when more information is available.

5. identified Aug 26, 2026, 12:30 PM UTC

   We have identified a mitigation for the Azure DevOps availability degradation in Europe and are currently applying it. We expect to complete the mitigation within the next 60 minutes and are monitoring its effectiveness. We will provide another update as progress is confirmed.

6. resolved Aug 26, 2026, 02:02 PM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 26, 2026, 06:31 AM UTC· resolved Aug 26, 2026, 03:45 PM UTC

1. identified Aug 26, 2026, 06:32 AM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Aug 26, 2026, 07:26 AM UTC

   Our engineers identified root cause of the issue and are working on a mitigation. We apologize for the inconvenience caused to our users.

3. identified Aug 26, 2026, 08:47 AM UTC

   Our engineers are working on the mitigation of the issue, unfortunately we can not provide timeline of the mitigation yet.

4. identified Aug 26, 2026, 09:59 AM UTC

   We’re investigating an issue that may cause intermittent delays or degraded performance when using Azure Artifacts in Southeast Asia. Our engineers are working to mitigate the issue and restore normal service. At this time, we cannot provide an estimated time to mitigation.

5. identified Aug 26, 2026, 11:07 AM UTC

   Our investigation into the Azure Artifacts issue in Southeast Asia is ongoing. Engineers are working to further understand the issue and reduce customer impact. We do not yet have a mitigation timeline and will provide another update when more information is available.

6. identified Aug 26, 2026, 12:30 PM UTC

   We have identified a mitigation for the Azure Artifacts issue in Southeast Asia and are currently applying it. We expect to complete the mitigation within the next 60 minutes and are monitoring its effectiveness. We will provide another update as progress is confirmed.

7. resolved Aug 26, 2026, 02:02 PM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 25, 2026, 04:09 PM UTC· resolved Aug 25, 2026, 05:25 PM UTC

1. resolved Aug 25, 2026, 05:55 PM UTC

   For the duration of the incident, NuGet operations were significantly impacted. The issue has been addressed, and NuGet operations are now completing successfully. Engineers are looking into the root cause. We apologize for the disruption.

Read the full incident report →

Started Aug 20, 2026, 03:30 PM UTC· resolved Aug 20, 2026, 04:24 PM UTC

1. identified Aug 20, 2026, 03:59 PM UTC

   We are currently investigating an issue affecting Azure DevOps Audit Log Streaming. Impacted customers may experience failures when streaming audit logs to external destinations. Other Azure DevOps services and audit capabilities remain available. Our engineering teams have identified the issue and are actively mitigating it by rolling back a recently introduced change associated with audit log streaming. We are assessing the full scope of impact and working to restore normal functionality as quickly as possible. We will continue to provide updates as more information becomes available.

2. resolved Aug 20, 2026, 04:29 PM UTC

   The issue affecting Azure DevOps Audit Log Streaming has been mitigated. A rollback of the offending change has completed across all impacted rings, and service functionality has been restored. We will continue monitoring to ensure service health

Read the full incident report →

Started Aug 18, 2026, 07:18 PM UTC· resolved Aug 19, 2026, 01:03 AM UTC

1. identified Aug 18, 2026, 07:53 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps involving sign in/out loops. The event is being triaged and we will post an update as soon as we know more.

2. identified Aug 18, 2026, 08:48 PM UTC

   Investigation showed customers are hitting distinct issues, mostly around tenant conditional access policies (CAP) triggering a known issue in Azure DevOps that may result in looping logins. Customers are encourged to close all tabs and clear their browser cache if they experience issues. Our engineers are disabling a feature flag to provide a broad mitigation to these issues.

3. identified Aug 18, 2026, 11:05 PM UTC

   The mitigation has been successful rolled out to all regions and our engineers are monitoring telemetry. Customers are encourged to close all tabs and clear their browser cache if they continue to experience issues.

4. resolved Aug 19, 2026, 01:05 AM UTC

   The mitigation has completed its rollout to all regions. Customers are encourged to close all tabs and clear their browser cache if they continue to experience issues. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 17, 2026, 04:07 PM UTC· resolved Aug 17, 2026, 06:12 PM UTC

1. identified Aug 17, 2026, 04:07 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. resolved Aug 17, 2026, 06:15 PM UTC

   The issue is now fully mitigated. Planned maintenance caused a database to become unresponsive, and SQL engineers performed a failover to restore. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 17, 2026, 01:26 PM UTC· resolved Aug 17, 2026, 02:05 PM UTC

1. identified Aug 17, 2026, 01:26 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. resolved Aug 17, 2026, 02:06 PM UTC

   The issue affecting Azure DevOps services in the East US 2 region has been mitigated. Customers may have experienced intermittent command failures and degraded service availability during the impact window. Our investigation identified a potential correlation with a recent configuration change rollout. As a mitigation step, the change was rolled back, and service health has returned to normal. We are continuing to closely monitor the service and conduct a detailed root cause analysis to better understand the underlying cause and prevent recurrence. Thank you for your patience while we worked to restore service.

Read the full incident report →

Started Aug 17, 2026, 05:41 AM UTC· resolved Aug 17, 2026, 12:46 PM UTC

1. identified Aug 17, 2026, 05:43 AM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Aug 17, 2026, 07:03 AM UTC

   We are currently investigating an issue affecting the Azure DevOps Marketplace/Gallery experience. Customers may experience difficulties accessing or loading the Marketplace Gallery homepage, and some Marketplace operations may be impacted. Our engineering teams are actively engaged in diagnosis and mitigation efforts. We understand the impact this may have on your workflows and are treating this issue with high priority. We will continue to provide regular updates as our investigation progresses and share additional details as they become available. Thank you for your patience while we work to restore normal service.

3. identified Aug 17, 2026, 07:59 AM UTC

   We continue to actively investigate the issue affecting the Azure DevOps Marketplace/Gallery experience. Customers may still experience difficulties accessing or loading the Marketplace Gallery homepage. Our engineering teams remain fully engaged in diagnosing the issue and identifying the safest path to mitigation. We understand the impact this may have on your workflows and appreciate your patience as we work toward resolution. We will continue to provide regular updates and share additional information as it becomes available.

4. identified Aug 17, 2026, 09:02 AM UTC

   Our engineering teams remain fully engaged in diagnosing the issue and identifying the safest path to mitigation. We understand the impact this may have on your workflows and appreciate your patience as we work toward resolution. We will continue to provide regular updates and share additional information as it becomes available.

5. identified Aug 17, 2026, 10:05 AM UTC

   We understand this incident has been ongoing longer than expected and want to assure customers that the investigation remains our highest priority. Engineering teams continue to actively analyze the issue and validate potential contributing factors to identify the safest and most effective mitigation. Customers may continue to experience difficulties accessing or loading the Azure DevOps Marketplace Gallery homepage. We recognize the disruption this may cause and appreciate your patience while we work through the investigation. Our teams are engaged around the clock and will continue to provide regular updates, even if there are no significant changes to report, so that customers remain informed of our progress. We will share our next update as soon as additional information becomes available.

6. identified Aug 17, 2026, 11:15 AM UTC

   Our engineering teams remain fully engaged in diagnosing the issue and identifying the safest path to mitigation. We understand the impact this may have on your workflows and appreciate your patience as we work toward resolution. We will continue to provide regular updates and share additional information as it becomes available.

7. resolved Aug 17, 2026, 12:46 PM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →

Started Aug 10, 2026, 04:45 PM UTC· resolved Aug 10, 2026, 04:59 PM UTC

1. identified Aug 10, 2026, 04:46 PM UTC

   Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.

2. identified Aug 10, 2026, 05:29 PM UTC

   Our engineers are working to understand and mitigate the issue.

3. resolved Aug 10, 2026, 06:59 PM UTC

   The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.

Read the full incident report →