Happydance experienced a minor incident on July 30, 2024 affecting Microsoft Azure, lasting 9h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 30, 2024, 12:17 PM UTC
We are currently investigating a possible ISP network DNS issue thats preventing some of our sites from loading for certain customer. The investigation is ongoing and updates will be provided every 30mins. There could be a related issue posted by Microsoft: Azure Portal and Service Status Page unavailable. We are investigating an issue impacting the Azure portal. More details will be provided as they become available. This message was last updated at 11:50 UTC on 30 July 2024
- identified Jul 30, 2024, 12:41 PM UTC
We believe the issue to be related to Azure Front Door and Edge routing and are waiting for an update from Microsoft. Symptoms are showing that across UK & Europe, websites hosted behind Azures premium WAF (Front Door) are having timeout issues, where the front door is unable to direct the request.
- monitoring Jul 30, 2024, 01:22 PM UTC
Update from Microsoft: Network Infrastructure - Issues accessing a subset of Microsoft services We are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services. We have multiple engineering teams engaged to diagnose and resolve the issue. More details will be provided as soon as possible. This message was last updated at 13:13 UTC on 30 July 2024
- monitoring Jul 30, 2024, 02:04 PM UTC
Microsoft Update: Network Infrastructure - Issues accessing a subset of Microsoft services Starting at approximately 12:00 UTC on 30 July 2024, a subset of customers may experience issues connecting to Microsoft services globally. Current status: We have multiple engineering teams engaged to diagnose and resolve the issue as soon as possible. We've identified multiple workstreams and are working to mitigate impacted workstreams by performing failover operations. More details will be provided as they become available. This message was last updated at 13:56 UTC on 30 July 2024 Since the issue is out of our control, we will now move to giving updates when received from Microsoft, not every 30 minutes.
- monitoring Jul 30, 2024, 02:50 PM UTC
Microsoft Update: Network Infrastructure - Issues accessing a subset of Microsoft services Starting approximately at 11:45 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to Microsoft services globally. Current Status: We have implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we are continuing to monitor to ensure full recovery. This message was last updated at 14:37 UTC on 30 July 2024 We continue to monitor this situation closely and will provide updates as we receive them from Microsoft
- monitoring Jul 30, 2024, 04:59 PM UTC
From the Microsoft Azure team: An unexpected usage spike resulted in Azure Front Door (AFD) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief. Our monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards, but we are investigating reports of specific services and regions that are still experiencing intermittent errors. We will provide an update on our continued mitigation efforts by 18:00 UTC, or sooner if we have progress to share.
- monitoring Jul 30, 2024, 05:57 PM UTC
From the Microsoft Azure Team: Current Status: An unexpected usage spike resulted in Azure Front Door (AFD) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief. Our monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards. As we investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some inadvertent side effects to certain services. We are updating our mitigation approach to minimize these side effects, and applying these following Safe Deployment Practices - beginning in Asia Pacific regions and then expanding in phases. We will provide an update on our continued mitigation efforts by 19:00 UTC, or sooner if we have progress to share.
- monitoring Jul 30, 2024, 06:28 PM UTC
From the Microsoft Azure Team: Impact Statement: Starting approximately at 11:45 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to Microsoft services globally. Current Status: An unexpected usage spike resulted in Azure Front Door (AFD) and Azure Content Delivery Network (CDN) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief. Our monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards. As we investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some side effects to certain services. We are updating our mitigation approach to minimize these side effects, and applying these following Safe Deployment Practices - beginning in Asia Pacific regions and then expanding in phases. We will provide an update on our continued mitigation efforts by 19:00 UTC, or sooner if we have progress to share.
- monitoring Jul 30, 2024, 09:09 PM UTC
From the Microsoft Azure team: Impact Statement: Starting approximately at 11:45 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to Microsoft services globally. An unexpected usage spike resulted in Azure Front Door (AFD) and Azure Content Delivery Network (CDN) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. Current Status: We implemented networking configuration changes and performed failovers to alternate networking paths to provide relief. We observed significant improvements in availability at 14:10 UTC, but continued to investigate and mitigate reports of specific services and regions that were still experiencing intermittent errors. While our initial network configuration changes successfully mitigated the impacts of the usage spike, these changes caused side effects for a small number of services. We proceeded with an updated mitigation approach, first rolling out across regions in Asia Pacific and Europe, and now applied to regions in the Americas. In line with our previous estimate of recovery by 20:30 UTC, the vast majority of customers and services are fully mitigated, and we are in the final stages of validating recovery. We will provide an update on our continued mitigation efforts by 21:00 UTC, or sooner if we have progress to share.
- resolved Jul 30, 2024, 09:26 PM UTC
This incident has been resolved.