Microsoft Major Outage - 365 Suite

Update timeline

1. identified Nov 26, 2024, 05:50 PM UTC

   ID: MO941162 Issue type: Incident Status Service Degradation Impacted services Microsoft 365 suite, Exchange Online, SharePoint Online, Microsoft Teams, Microsoft Power Automate in Microsoft 365, Universal Print, Microsoft Purview Details Title: Users may be unable to access or use some Microsoft 365 services and features User impact: Users may be unable to access or use some Microsoft 365 services and features. More info: The affected scenarios are as follows: Users may be unable to access Exchange Online using the following impacted connection methods: - Outlook on the web - Outlook desktop client - Representational State Transfer (REST) - Exchange ActiveSync (EAS) Users may be unable to use the following features within Microsoft Teams: - Users are unable to create or update Virtual Events, including webinars and Town Halls. - Users may be unable to access or modify their calendar in Microsoft Teams. This would include loading calendar, viewing meetings, creating/updating meetings and joining meetings. - Users are unable to create chat, add users and create or edited meetings. - Users are unable to create or modify new teams and channels. - Users may be unable to update presence. - Users may be unable to use the search function. - Users may not see updated list of files and links failing to load within the Chat shared tab. Users may experience the following issues with Microsoft Purview: - Users may be unable to access the Purview Portal, or Purview Solutions. - Users may experience delays in policy stamping and with Adaptive Scope Evaluations. Users may be unable to export content or set and view labels within Microsoft Fabric. Some Microsoft Fabric users with Purview Information Protection Policies with sensitivity labels enabled, may be unable to use interactive operations on Power BI Desktop format files and reports, including export operations on Fabric artifacts with Sensitivity labels applied. Users may be unable to use the search feature within SharePoint Online. Users may be unable to perform the following actions within Microsoft Defender for Office365. - Users may be unable to create simulations, simulation payloads or end user notifications. - Users may experience issues with delivery for end user notifications and simulation messages - Users may experiences issues with viewing simulation reports, and content. Users may also be unable to print via Universal Print. Users may experience errors running flows that utilize cloud connectors in Power Automate for Desktop. Current status: We’ve identified a recent change which we believe has resulted in impact. We’ve begun to revert the change and are investigating what additional actions are required to mitigate the issue. Scope of impact: Impact is specific to users who are served through the affected infrastructure. Start time: Sunday, November 24, 2024, at 1:20 AM UTC Preliminary root cause: A portion of infrastructure which supports mailbox and calendar functionality isn't operating as expected, resulting in impact. Next update by: Monday, November 25, 2024, at 2:00 PM UTC --------------------------------------------------

2. identified Nov 26, 2024, 05:51 PM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Nov 26, 2024, 05:54 PM UTC

   Current status Nov 26, 2024, 7:53 AM EST We’re continuing our period of monitoring service telemetry, which shows the service availability has remained healthy. Next update by: Tuesday, November 26, 2024 at 11:00 AM EST -------------------------- Users may be unable to access or use some Microsoft 365 services and features Issue ID: MO941162 Affected services: Exchange Online, Microsoft 365 suite, Microsoft Bookings, Microsoft Copilot (Microsoft 365), Microsoft Power Automate in Microsoft 365, Microsoft Purview, Microsoft Teams, SharePoint Online, Universal Print Status: Service degradation Issue type: Incident Start time: Nov 24, 2024, 8:20 PM EST More info The impacted services and their impact are as follows: Exchange Online - Users may be unable to access using the following impacted connection methods: Outlook on the web, Outlook desktop client, Representational State Transfer (REST), Exchange ActiveSync (EAS) - Users may experience mail transport delays. Microsoft Teams - Users are unable to create or update Virtual Events, including webinars and Town Halls. - Users may be unable to access or modify their calendar in Microsoft Teams. This would include loading the calendar, viewing meetings, creating/updating meetings and joining meetings. - Users are unable to create chat, add users and create or edited meetings. - Users are unable to create or modify new teams and channels. - Users may be unable to update presence. - Users may be unable to use the search function. - Users may not see an updated list of files and links failing to load within the Chat shared tab. Microsoft Purview - Users may be unable to access the Purview Portal, or Purview Solutions. - Users may experience delays in policy stamping and with Adaptive Scope Evaluations. Microsoft Fabric - Users may be unable to export content or set and view labels. - Some Microsoft Fabric users with Purview Information Protection Policies with sensitivity labels enabled, may be unable to use interactive operations on Power BI Desktop format files and reports, including export operations on Fabric artifacts with Sensitivity labels applied. SharePoint Online - Users may be unable to use the search feature. Microsoft Defender for Office365 - Users may be unable to create simulations, simulation payloads or end user notifications. - Users may experience issues with delivery for end user notifications and simulation messages. - Some users may experience failures in manual or AIR approved Remediation Actions submitted through ThreatExplorer, Advanced Hunting or the Action Center. - Users may experience issues with viewing simulation reports, and content. - Users may get a “You can’t access this section” error when accessing sections of the Defender XDR portal, such as the Incidents and Alerts pages, that include affected Defender for Office 365 shared components. Universal Print - Users may be unable to Print via Universal Print. - Users may be unable to list Printers/Printer Shares on the Azure Portal Universal Print blade. - Users may be unable to Register Printers via Universal Print. Power Automate for Desktop - Users may experience errors when running flows that utilize cloud connectors. Microsoft Bookings - Users may be unable to access their bookings. Microsoft Viva Engage - Users may be unable to load topics in feeds and threads. - Users may see searches or autocomplete fail for topic experiences. - Users’ Microsoft Teams Q&A experiences may not show user names. - Users may not be able to load their home feeds. Microsoft Copilot - Users are unable to use the personal Copilot panel in meetings and post meetings. - Users are unable to see historic Copilot conversation history in meetings and post meetings. - Users may not be able to load Copilot experiences within the Viva Engage service. - Users may be unable to submit queries to Copilot within OneNote through the chat pane and context menu. Users may receive an apology or "at capacity" message." Scope of impact This issue could impact any user hosted globally; however, users are experiencing varying degrees of impact depending on how they are routed through the affected infrastructure and what action they are trying to perform. Impacted users attempting to use the functionalities outlined in the More info section of this communication may be affected by this event. Preliminary root cause Due to a recent change that decommissioned a backend service, requests were directed to an incorrect endpoint. This resulted in request handling issues and ultimately affected machines' processing capabilities, which led to impact.

4. resolved Nov 26, 2024, 08:04 PM UTC

   Users may be unable to access or use some Microsoft 365 services and features ID: MO941162 Issue type: Incident Status Service Restored Impacted services Microsoft 365 suite Details Title: Users may be unable to access or use some Microsoft 365 services and features User impact: Users may be unable to access or use some Microsoft 365 services and features. More info: The impacted services and their impact scenarios were as follows: Exchange Online - Users may have been unable to access using the following impacted connection methods: Outlook on the web, Outlook desktop client, Representational State Transfer (REST), Exchange ActiveSync (EAS) - Users may have experienced mail transport delays. Microsoft Teams - Users may have been unable to create or update Virtual Events, including webinars and Town Halls. - Users may have been unable to access or modify their calendar in Microsoft Teams. This would include loading the calendar, viewing meetings, creating/updating meetings and joining meetings. - Users may have been unable to create chat, add users and create or edited meetings. - Users may have been unable to create or modify new teams and channels. - Users may have been unable to update presence. - Users may have been unable to use the search function. - Users may not have seen an updated list of files and links failing to load within the Chat shared tab. Microsoft Purview - Users may have been unable to access the Purview Portal, or Purview Solutions. - Users may have experienced delays in policy stamping and with Adaptive Scope Evaluations. Microsoft Fabric - Users may have been unable to export content or set and view labels. - Some Microsoft Fabric users with Purview Information Protection Policies with sensitivity labels enabled, may have been unable to use interactive operations on Power BI Desktop format files and reports, including export operations on Fabric artifacts with Sensitivity labels applied. SharePoint Online - Users may have been unable to use the search feature. Microsoft Defender for Office365 and the XDR portal - Users may have been unable to create simulations, simulation payloads or end user notifications. - Users may have experienced issues with delivery for end user notifications and simulation messages. - Some users may have experienced failures in manual or AIR approved Remediation Actions submitted through ThreatExplorer, Advanced Hunting or the Action Center. - Users may have experienced issues with viewing simulation reports, and content. - Users may have received a “You can’t access this section” error when accessing sections of the Defender XDR portal, such as the Incidents and Alerts pages, that include affected Defender for Office 365 shared components. Universal Print - Users may have been unable to Print via Universal Print. - Users may have been unable to list Printers/Printer Shares on the Azure Portal Universal Print blade. - Users may have been unable to Register Printers via Universal Print. Power Automate for Desktop - Users may have experienced errors when running flows that utilize cloud connectors. Microsoft Bookings - Users may have been unable to access their bookings. Microsoft Viva Engage - Users may have been unable to load topics in feeds and threads. - Users may have seen searches or autocomplete fail for topic experiences. - Users’ Microsoft Teams Q&A experiences may not have shown user names. - Users may not have been able to load their home feeds. Microsoft Copilot - Users may have been unable to use the personal Copilot panel in meetings and post meetings. - Users may have been unable to see historic Copilot conversation history in meetings and post meetings. - Users may have been unable to load Copilot experiences within the Viva Engage service. - Users may have been unable to submit queries to Copilot within OneNote through the chat pane and context menu. Users may have received an apology or "at capacity" message. Final Status: After a period of extended monitoring, we’ve confirmed through customer reports and service health telemetry that the issue is resolved. Scope of impact: This issue could have impacted any user hosted globally; however, users experienced varying degrees of impact depending on how they were routed through the affected infrastructure and what action they were trying to perform. Impacted users who attempted to use the functionalities outlined in the More info section of this communication may have been affected by this event Start time: Sunday, November 24, 2024, at 1:20 AM UTC End time: Tuesday, November 26, 2024, at 11:00 AM UTC Preliminary root cause: Due to a recent change that decommissioned a backend service, requests were directed to an incorrect endpoint. This resulted in request handling issues and affected servers' processing capabilities, which led to impact. Next steps: - We're examining the parameters required to decommission backend services so we can better anticipate, test for, and avoid or prevent similar scenarios. - We're assessing monitoring optimizations we can better detect and more quickly remediate router service issues. We'll provide a preliminary Post-Incident Report within two business days and a final Post-Incident Report within five business days.