Resolved after 2 hours 55 minutes, affecting Core services (United States) and Core services (Canada) and 1 more, on August 31, 2026.
On August 31, 2026, between 15:50 and 18:45 UTC, a subset of Azure DevOps customers globally experienced intermittent failures and elevated latency across core workflows including Boards, Repos, Pipelines, Test Plans, Artifacts, Analytics, and Authentication. We understand how disruptive outages can be, especially when they affect multiple services at once, and sincerely apologize to all impacted customers. ## Background Azure DevOps uses Azure Front Door (AFD) Web Application Firewall (WAF) to help protect the service from malicious traffic. WAF rules can run in audit mode, where matching requests are logged, or prevention mode, where matching requests are blocked. Before enforcing new rules broadly, we rely on recent representative audit data, service telemetry, deployment validation, and review gates to confirm that rules do not incorrectly match and block legitimate traffic. As part of ongoing security hardening, a curated subset of Microsoft Default Rule Set 2.0 rules was being enabled in prevention mode across our services by updating WAF configurations for affected Azure Front Door profiles. ## What triggered the outage? In the month of August, we began rolling out WAF configuration changes to all services, with each change taking 30 minutes to reach all AFD edge nodes globally. The WAF configuration change was triggered on the last batch of services between 15:21 and 15:48 UTC. At 15:50 UTC, a class of legitimate customer requests unexpectedly started being blocked with HTTP 403 responses. At 16:07 UTC our monitoring system started alerting us of customer impact and our engineering team quickly engaged and started investigating. At 16:17 UTC, additional legitimate requests began failing in an internal Azure DevOps service-to-service path. HTTP 403 responses returned with connection-close behavior, and due to the high volume of internal calls, repeated connection closures temporarily exhausted Source Network Address Translation (SNAT) ports in some of the Software Load Balancer (SLB) instances used by the calling service. When those ports were exhausted, the calling service could not reliably open outbound connections to dependent services, which surfaced to customers as HTTP 500 and 503 responses across a broader set of endpoints. Once the engineering team identified the WAF configuration changes as the very likely cause, we immediately started to roll them back. The first rollback configuration was created at 17:17 UTC and reduced part of the direct WAF-blocking impact. It did not fully recover the service because we had not yet identified that one internal service-to-service path was driving SNAT port exhaustion and amplifying failures across additional endpoints. We rolled back that service’s WAF configuration at 18:12 UTC, and recovery completed as the rollback propagated through AFD edge nodes by 18:45 UTC. After investigation, we discovered that the representative audit data used for validation did not include the period when the relevant rules were running in audit mode. ## What contributed to the mitigation time? Several factors increased the time required to identify and mitigate the issue: - The rollout was regionally global and applied to shared Azure Front Door configurations, so a single WAF configuration change could affect customers in multiple regions at once. - The initial false-positive block presented as both direct HTTP 403 failures and downstream symptoms across Boards, Repos, Pipelines, Test Plans, Artifacts, Analytics, and Authentication, which made it harder to separate service-local failures from the traffic-filtering issue. - Operational signals were fragmented. Early service-specific alerts and support cases did not immediately correlate with the WAF rollout, and near real-time WAF logs and monitoring for first-party traffic were insufficient to quickly identify the cause. ## Next Steps - We have reverted the problematic rule deployment and will not expand our WAF rule set in prevention mode until we harden our validation to demonstrate they do not block legitimate Azure DevOps traffic. - We are strengthening pre-deployment validation for WAF changes by requiring recent, representative audit-mode analysis, explicit architecture and security review, and guardrails that block enforcement when coverage is incomplete or stale. - We are improving telemetry and monitoring so WAF behavior, customer-impact alerts, service health signals, and support-case patterns can be correlated more quickly with Azure DevOps service health. - We are adopting staged rollouts for global and shared service resources, telemetry-based progression gates, and longer validation time between rollout stages. - We are improving rollback readiness so critical global configuration changes have pre-approved, tested rollback paths that do not depend on the failing component and reduce decision time during an incident. - We are improving our ability to troubleshoot SNAT port exhaustion and making changes to greatly reduce the likelihood of getting into a SNAT port exhaustion situation. - We are using this incident to improve how we validate, stage, monitor, and roll back global traffic-filtering changes. Sincerely, Mauro Ottaviani, Principal Group SWE Manager, Azure DevOps Engineering
Affected components
Update timeline
- identified Aug 31, 2026, 04:09 PM UTC
Our engineers are currently investigating an event impacting Azure DevOps. The event is being triaged and we will post an update as soon as we know more.
- identified Aug 31, 2026, 05:50 PM UTC
Our engineers are working to understand and mitigate the issue.
- identified Aug 31, 2026, 06:23 PM UTC
We have identified the change in configuration which triggered this incident. We are reverting the change and observing impact. Will post more updates on status, root-cause and learnings/repairs from this incident.
- identified Aug 31, 2026, 07:22 PM UTC
We have reverted the change and services are recovering. We will continue to observe the impact and post more updates on status.
- identified Aug 31, 2026, 07:56 PM UTC
The issue is now fully mitigated. Our engineers will be investigating this further to learn from and reduce the risk of potential recurrences. We apologize for the impact this had on our customers.
- identified Sep 02, 2026, 07:29 PM UTC
On August 31, 2026, beginning at 16:07 UTC, customers in multiple regions experienced intermittent failures and elevated latency across Azure DevOps services, including Boards, Repos, Pipelines, Test Plans, and Artifacts. The incident was triggered by the rollout of Azure Front Door Web Application Firewall (WAF) rules designed to enhance the security posture of Azure DevOps. An issue with the configuration caused legitimate traffic to be incorrectly blocked, leading to service degradation and failures across multiple services. During the incident, approximately 0.7% of Azure DevOps commands failed and approximately 0.1% of commands experienced elevated latency. The issue was mitigated by reverting the change and restoring the last known good configuration, and service was fully restored by 19:56 UTC. We are completing a full postmortem and will use its findings to strengthen our SDP and change-rollout review processes, including validation, monitoring, and rollback safeguards to reduce the risk of recurrence. The full postmortem and associated repair items will be published once complete.
- resolved Sep 24, 2026, 05:37 AM UTC
On August 31, 2026, between 15:50 and 18:45 UTC, a subset of Azure DevOps customers globally experienced intermittent failures and elevated latency across core workflows including Boards, Repos, Pipelines, Test Plans, Artifacts, Analytics, and Authentication. We understand how disruptive outages can be, especially when they affect multiple services at once, and sincerely apologize to all impacted customers. ## Background Azure DevOps uses Azure Front Door (AFD) Web Application Firewall (WAF) to help protect the service from malicious traffic. WAF rules can run in audit mode, where matching requests are logged, or prevention mode, where matching requests are blocked. Before enforcing new rules broadly, we rely on recent representative audit data, service telemetry, deployment validation, and review gates to confirm that rules do not incorrectly match and block legitimate traffic. As part of ongoing security hardening, a curated subset of Microsoft Default Rule Set 2.0 rules was being enabled in prevention mode across our services by updating WAF configurations for affected Azure Front Door profiles. ## What triggered the outage? In the month of August, we began rolling out WAF configuration changes to all services, with each change taking 30 minutes to reach all AFD edge nodes globally. The WAF configuration change was triggered on the last batch of services between 15:21 and 15:48 UTC. At 15:50 UTC, a class of legitimate customer requests unexpectedly started being blocked with HTTP 403 responses. At 16:07 UTC our monitoring system started alerting us of customer impact and our engineering team quickly engaged and started investigating. At 16:17 UTC, additional legitimate requests began failing in an internal Azure DevOps service-to-service path. HTTP 403 responses returned with connection-close behavior, and due to the high volume of internal calls, repeated connection closures temporarily exhausted Source Network Address Translation (SNAT) ports in some of the Software Load Balancer (SLB) instances used by the calling service. When those ports were exhausted, the calling service could not reliably open outbound connections to dependent services, which surfaced to customers as HTTP 500 and 503 responses across a broader set of endpoints. Once the engineering team identified the WAF configuration changes as the very likely cause, we immediately started to roll them back. The first rollback configuration was created at 17:17 UTC and reduced part of the direct WAF-blocking impact. It did not fully recover the service because we had not yet identified that one internal service-to-service path was driving SNAT port exhaustion and amplifying failures across additional endpoints. We rolled back that service’s WAF configuration at 18:12 UTC, and recovery completed as the rollback propagated through AFD edge nodes by 18:45 UTC. After investigation, we discovered that the representative audit data used for validation did not include the period when the relevant rules were running in audit mode. ## What contributed to the mitigation time? Several factors increased the time required to identify and mitigate the issue: - The rollout was regionally global and applied to shared Azure Front Door configurations, so a single WAF configuration change could affect customers in multiple regions at once. - The initial false-positive block presented as both direct HTTP 403 failures and downstream symptoms across Boards, Repos, Pipelines, Test Plans, Artifacts, Analytics, and Authentication, which made it harder to separate service-local failures from the traffic-filtering issue. - Operational signals were fragmented. Early service-specific alerts and support cases did not immediately correlate with the WAF rollout, and near real-time WAF logs and monitoring for first-party traffic were insufficient to quickly identify the cause. ## Next Steps - We have reverted the problematic rule deployment and will not expand our WAF rule set in prevention mode until we harden our validation to demonstrate they do not block legitimate Azure DevOps traffic. - We are strengthening pre-deployment validation for WAF changes by requiring recent, representative audit-mode analysis, explicit architecture and security review, and guardrails that block enforcement when coverage is incomplete or stale. - We are improving telemetry and monitoring so WAF behavior, customer-impact alerts, service health signals, and support-case patterns can be correlated more quickly with Azure DevOps service health. - We are adopting staged rollouts for global and shared service resources, telemetry-based progression gates, and longer validation time between rollout stages. - We are improving rollback readiness so critical global configuration changes have pre-approved, tested rollback paths that do not depend on the failing component and reduce decision time during an incident. - We are improving our ability to troubleshoot SNAT port exhaustion and making changes to greatly reduce the likelihood of getting into a SNAT port exhaustion situation. - We are using this incident to improve how we validate, stage, monitor, and roll back global traffic-filtering changes. Sincerely, Mauro Ottaviani, Principal Group SWE Manager, Azure DevOps Engineering
Your product runs on more than Azure DevOps
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services