Probado experienced a major incident on July 19, 2024 affecting Office 365: Email and Office 365: Teams, lasting 3h 40m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jul 19, 2024, 07:46 AM UTC
We're aware that some users may be impacted by a Microsoft service outage and are monitoring the situation. This is largely affecting offerings provided by microsoft directly and Probado hosted services are unaffected (Hosted Servers, VoIP Telephony etc) Microsoft Incident IDs: MO821132, EX820374
- monitoring Jul 19, 2024, 10:23 AM UTC
Update from MS: We're conducting additional mitigation actions to remediate impact for the remaining affected Microsoft 365 apps and services. Our telemetry indicates functionality and availability are continuing to improve across multiple scenarios, and we're monitoring this closely to ensure we're progressing towards full recovery. This quick update is designed to give the latest information on this issue.
- monitoring Jul 19, 2024, 11:03 AM UTC
Providing additional details on this issue: It appears the root cause of this Microsoft issue is with product vendor Crowdstrike who offer security solutions to Microsoft services. A corrupt update published through their software appears to of triggered a blue screen on Windows PCs and Servers. As previously mentioned both vendors are rolling out remediation updates at present and we're seeing affected services come back online.
- monitoring Jul 19, 2024, 11:20 AM UTC
11:24 Latest from MS: The underlying cause of the issue has been fixed and several Microsoft 365 apps and services have been restored to full functionality. Residual impact is still affecting some Microsoft 365 apps and services, and Microsoft 365 engineering are continuing to conduct additional mitigation actions to provide relief. We're continuing to observe an increase in functionality and availability for the remaining impacted scenarios and we're monitoring this closely to ensure we're progressing towards full recovery. Microsoft is continuing to treat this event with the highest possible priority.
- resolved Jul 19, 2024, 11:26 AM UTC
Microsoft has confirmed that the majority of services have been restored to full functionality. We'll continue to monitor and relay the root cause analysis of the incident when available.
- postmortem Jul 19, 2024, 11:27 AM UTC
**Services Affected:** * Microsoft Azure * Microsoft Defender * Microsoft Defender for Endpoint * Microsoft Defender Experts * Microsoft Fabric * Microsoft Intune * Microsoft OneNote * Microsoft Teams * OneDrive for Business * Microsoft PowerBI * SharePoint Online * Windows 365 * Viva Engage * Microsoft Purview **Scope of impact** This issue may be impacting any user attempting to use various Microsoft 365 apps and services. **Preliminary root cause** A configuration change in a portion of our Azure backend workloads, caused interruption between storage and compute resources which resulted in connectivity failures that affected downstream Microsoft 365 services dependent on these connections. \(NB: Post mortem subject to amendment as final version comes in\)