Microsoft Services Issue (MO842351)

Update timeline

1. identified Jul 30, 2024, 03:59 PM UTC

   We're aware some users may be experiencing difficulties connecting to 365 services. Microsoft are aware of the issue and are investigating: - - From Microsoft: We're aware of users experiencing intermittent connectivity issues with various 365 services: These lists will change as we confirm that different service are or aren’t affected. - Impacted services include, but are not limited to: - Microsoft 365 admin center - Intune - Entra - Power Platform - Power BI - Initial indications are that the following services are not impacted: - SharePoint Online - OneDrive for Business - Microsoft Teams - Exchange Online - - We'll update this notice when we receive more information.

2. monitoring Jul 30, 2024, 04:03 PM UTC

   We've implemented a networking configuration change, and some Microsoft 365 services have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we're continuing to monitor to ensure full recovery.

3. resolved Jul 31, 2024, 07:54 AM UTC

   This incident has been resolved.

4. postmortem Aug 01, 2024, 10:14 AM UTC

   Microsoft Interruption of Services Impacted services affected: - - Microsoft 365 admin center - Azure Services - Intune - Entra - Power Platform - Power BI Microsoft have concluded their report and identified that malicious actors attacked their infrastructure with a DDoS based attack. due to a configuration issue, technologies designed to mitigate against such attacks actually contributed to the degradation in performance. Below is an excerpt from Microsofts report \(Tracking ID: KTY1-HW8 [Azure status history | Microsoft Azure](https://azure.status.microsoft/en-us/status/history/)\) > _**What happened?**_ > > _Between approximately at 11:45 UTC and 19:43 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to a subset of Microsoft services globally. Impacted services included Azure App Services, Application Insights, Azure IoT Central, Azure Log Search Alerts, Azure Policy, as well as the Azure portal itself and a subset of Microsoft 365 and Microsoft Purview services._ > > _**What do we know so far?**_ > > _An unexpected usage spike resulted in Azure Front Door \(AFD\) and Azure Content Delivery Network \(CDN\) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. While the initial trigger event was a Distributed Denial-of-Service \(DDoS\) attack, which activated our DDoS protection mechanisms, initial investigations suggest that an error in the implementation of our defenses amplified the impact of the attack rather than mitigating it._ > > _**How did we respond?**_ > > _Customer impact began at 11:45 UTC and we started investigating. Once the nature of the usage spike was understood, we implemented networking configuration changes to support our DDoS protection efforts, and performed failovers to alternate networking paths to provide relief. Our initial network configuration changes successfully mitigated majority of the impact by 14:10 UTC. Some customers reported less than 100% availability, which we began mitigating at around 18:00 UTC. We proceeded with an updated mitigation approach, first rolling this out across regions in Asia Pacific and Europe. After validating that this revised approach successfully eliminated the side effect impacts of the initial mitigation, we rolled it out to regions in the Americas. Failure rates returned to pre-incident levels by 19:43 UTC - after monitoring traffic and services to ensure that the issue was fully mitigated, we declared the incident mitigated at 20:48 UTC. Some downstream services took longer to recover, depending on how they were configured to use AFD and/or CDN._ > > _**What happens next?**_ > > _Our team will be completing an internal retrospective to understand the incident in more detail. We will publish a Preliminary Post Incident Review \(PIR\) within approximately 72 hours, to share more details on what happened and how we responded. After our internal retrospective is completed, generally within 14 days, we will publish a Final Post Incident Review with any additional details and learnings._ > While this issue was considered MBOC for Service Providers around the globe. We can state that because we host a Private cloud environment our customer base was largely protected from this outage as their server infrastructure is not Azure based.