Monkey Tree Hosting incident

Unable to access multiple Microsoft 365 services

Minor Resolved View vendor source →

Monkey Tree Hosting experienced a minor incident on July 30, 2024, lasting 8h 49m. The incident has been resolved; the full update timeline is below.

Started
Jul 30, 2024, 12:24 PM UTC
Resolved
Jul 30, 2024, 09:13 PM UTC
Duration
8h 49m
Detected by Pingoru
Jul 30, 2024, 12:24 PM UTC

Update timeline

  1. investigating Jul 30, 2024, 12:24 PM UTC

    Microsoft are investigating a potential issue and checking for impact. An update will be posted within 30 minutes..

  2. identified Jul 30, 2024, 01:14 PM UTC

    Microsoft have identified a potential networking issue and are investigating this further..

  3. identified Jul 30, 2024, 01:47 PM UTC

    Microsoft have confirmed that the issue is only occurring within a section of their networking infrastructure and doesn't impact all Microsoft 365 services. Microsoft are working on a list of affecting services..

  4. identified Jul 30, 2024, 02:18 PM UTC

    These lists will change as we confirm that different services are or aren’t affected. Impacted services include, but are not limited to: \- Microsoft 365 admin center \- Intune \- Entra \- Power Platform Initial indications are that the following services are not impacted: \- SharePoint Online \- OneDrive for Business \- Microsoft Teams \- Exchange Online.

  5. identified Jul 30, 2024, 02:19 PM UTC

    Microsoft are analyzing traffic patterns within a section of a networking infrastructure to assist our investigations. Additionally, they are reviewing mitigation options, including potential failovers, to provide relief..

  6. monitoring Jul 30, 2024, 02:46 PM UTC

    Microsoft have implemented a networking configuration change, and some Microsoft 365 services have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we're continuing to monitor to ensure full recovery..

  7. monitoring Jul 30, 2024, 03:51 PM UTC

    Telemetry shows that the service has remained stable. We're continuing to monitor the service for an extended period of time to confirm resolution..

  8. identified Jul 30, 2024, 04:52 PM UTC

    Microsoft have isolated a small amount of network packet drops and residual impact. They are adjusting the networking configuration changes to mitigate impact. Availability for most services remains stable and we continue to monitor to ensure full mitigation..

  9. monitoring Jul 30, 2024, 06:24 PM UTC

    As Microsoft investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some inadvertent side effects to certain services. Microsoft are updating our mitigation approach to minimize these side effects and applying these changes in phases to ensure no additional impact is observed..

  10. monitoring Jul 30, 2024, 07:23 PM UTC

    Microsoft have confirmed that the updated mitigation approach is successfully addressing the remaining impact. We're applying these changes in the final regions and will continue to monitor service health to ensure the issue is fully resolved once completed..

  11. monitoring Jul 30, 2024, 08:17 PM UTC

    Microsoft have completed our additional mitigation actions to address the adverse impact. We're now performing final validations to confirm the issue is fully resolved..

  12. resolved Jul 30, 2024, 09:13 PM UTC

    Microsoft have confirmed through extending monitoring and internal validations that the issue is now resolved..