Block 64 incident
Combine Analytics Portal, API and Support Portal Partial Outage
Block 64 experienced a major incident on July 30, 2024 affecting Combine Analytics Portal and Combine API and 1 more component, lasting 8h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 30, 2024, 01:05 PM UTC
Our web services are currently experiencing a partial outage as certain users are experiencing connectivity issues . Our support teams are investigating the issue.
- investigating Jul 30, 2024, 01:17 PM UTC
Our infrastructure provider Azure is currently experiencing a widespread Network Infrastructure outage which is impacting these services. Please see their latest update below: Warning Network Infrastructure - Issues accessing a subset of Microsoft services We are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services. We have multiple engineering teams engaged to diagnose and resolve the issue. More details will be provided as soon as possible. This message was last updated at 13:13 UTC on 30 July 2024 Azure Status: https://azure.status.microsoft/en-us/status
- investigating Jul 30, 2024, 01:59 PM UTC
Latest Azure Update: Network Infrastructure - Issues accessing a subset of Microsoft services Starting at approximately 12:00 UTC on 30 July 2024, a subset of customers may experience issues connecting to Microsoft services globally. Current status: We have multiple engineering teams engaged to diagnose and resolve the issue as soon as possible. We've identified multiple workstreams and are working to mitigate impacted workstreams by performing failover operations. More details will be provided as they become available. This message was last updated at 13:56 UTC on 30 July 2024 We will continue to monitor and provide updates as they become available.
- identified Jul 30, 2024, 02:02 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 30, 2024, 02:41 PM UTC
Latest Azure Update: Warning Network Infrastructure - Issues accessing a subset of Microsoft services Starting approximately at 11:45 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to Microsoft services globally. Current Status: We have implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we are continuing to monitor to ensure full recovery. This message was last updated at 14:37 UTC on 30 July 2024
- monitoring Jul 30, 2024, 02:59 PM UTC
Network Infrastructure - Issues accessing a subset of Microsoft services Starting approximately at 11:45 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to Microsoft services globally. Current Status: We have implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards, and we are continuing to monitor to ensure full recovery. This message was last updated at 14:54 UTC on 30 July 2024
- monitoring Jul 30, 2024, 04:05 PM UTC
Performance does appear to have stabilized. That said, the status from Azure remains the same: "implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards, and we are continuing to monitor to ensure full recovery." We will continue to monitor the situation.
- monitoring Jul 30, 2024, 04:45 PM UTC
Azure is reporting performance issues and intermittent errors and timeouts due to usage spikes. Network Infrastructure - Issues accessing a subset of Microsoft services Impact statement: Starting approximately at 11:45 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to Microsoft services globally. Current Status: An unexpected usage spike resulted in Azure Front Door (AFD) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief. Our monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards, but we are investigating reports of specific services and regions that are still experiencing intermittent errors. We will provide an update on our continued mitigation efforts by 18:00 UTC, or sooner if we have progress to share. This message was last updated at 16:42 UTC on 30 July 2024 We will continue to monitor and provide updates as they become available.
- monitoring Jul 30, 2024, 06:00 PM UTC
Azure is reporting some inadvertent side effects to certain services and some services / usages are still affected: Network Infrastructure / Azure Front Door - Issues accessing a subset of Microsoft services Impact Statement: Starting approximately at 11:45 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to Microsoft services globally. Current Status: An unexpected usage spike resulted in Azure Front Door (AFD) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief. Our monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards. As we investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some inadvertent side effects to certain services. We are updating our mitigation approach to minimize these side effects, and applying these following Safe Deployment Practices - beginning in Asia Pacific regions and then expanding in phases. We will provide an update on our continued mitigation efforts by 19:00 UTC, or sooner if we have progress to share. This message was last updated at 17:56 UTC on 30 July 2024
- monitoring Jul 30, 2024, 07:02 PM UTC
Latest update from Azure: An unexpected usage spike resulted in Azure Front Door (AFD) and Azure Content Delivery Network (CDN) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief. Our monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards. We continue to investigate and mitigate reports of specific services and regions that are still experiencing intermittent errors. While our previous network configuration changes successfully mitigated the impacts of the usage spike, these changes caused side effects for a small number of services. Our updated mitigation approach has successfully reduced these impacts across regions in Asia Pacific and Europe. Our phased approach is now applying this same mitigation targeting regions in the Americas, and we expect that services should be fully mitigated by 20:30 UTC. We will provide an update on our continued mitigation efforts by 20:00 UTC, or sooner if we have progress to share. This message was last updated at 19:00 UTC on 30 July 2024
- monitoring Jul 30, 2024, 08:17 PM UTC
Latest Azure Update: We implemented networking configuration changes and performed failovers to alternate networking paths to provide relief. We observed significant improvements in availability at 14:10 UTC, but continued to investigate and mitigate reports of specific services and regions that were still experiencing intermittent errors. While our initial network configuration changes successfully mitigated the impacts of the usage spike, these changes caused side effects for a small number of services. We proceeded with an updated mitigation approach, first rolling out across regions in Asia Pacific and Europe, and now applied to regions in the Americas. In line with our previous estimate of recovery by 20:30 UTC, the vast majority of customers and services are fully mitigated, and we are in the final stages of validating recovery. We will provide an update on our continued mitigation efforts by 21:00 UTC, or sooner if we have progress to share. This message was last updated at 20:06 UTC on 30 July 2024
- resolved Jul 30, 2024, 09:14 PM UTC
Azure has restored full service and resolved the underlying issues.