Mesh Security experienced a notice incident on November 25, 2024 affecting Inbound Email Security and Inbound Email Security, lasting 16h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 25, 2024, 08:15 AM UTC
We are currently investigating an issue where mails are slow to process on the Mesh o365 service. Unified users will also be impacted at the o365 level. Gateway users are unaffected.
- identified Nov 25, 2024, 08:57 AM UTC
We believe the issue is related to an ongoing incident (Issue ID: MO941162) within o365 that is affecting our ability to retrieve messages via the Graph API. Messages are still slow to process on the 365 level of the Mesh service. Gateway users are unaffected.
- monitoring Nov 25, 2024, 09:48 AM UTC
We're continuing to monitor our queues and the current Microsoft Incident for updates. We've reached out to our internal contacts within MS for further updates. Mails are still slow to process on the o365 components of the Mesh service in the EU. Unified users will also be impacted at the o365 level. Gateway users are unaffected.
- monitoring Nov 25, 2024, 02:00 PM UTC
There are queues related to this issue now on the US infrastructure. Some mail is slow to process on the Mesh o365 service. Unified users will also be impacted at the o365 level. Gateway users are unaffected. Queues on the EU infrastructure have been stabilising and we expect them to clear in the next 30 minutes.
- monitoring Nov 25, 2024, 02:17 PM UTC
EU queue levels have returned to normal levels. On the US Mails are still slow to process on the Mesh 365 service. Unified users will also be impacted at the O365 level. Gateway users are unaffected. The latest update from O365 is that they're deploying rolling fixes. We'll continue to monitor the US infrastructure and provide updates.
- monitoring Nov 25, 2024, 03:29 PM UTC
In their latest update, Microsoft state that they pushed a fix to 60% of the environments affected. The current impact to customers on Mesh is only on our US infrastructure and only applies to Mesh 365 and the API component of Mesh Unified. Mesh Gateway and the gateway component of Mesh Unified are not impacted. Microsoft are posting updates here: https://admin.microsoft.com/#/homepage/:/alerts/MO941162 and on X - https://x.com/MSFT365Status/status/1860973220662280356
- monitoring Nov 25, 2024, 06:57 PM UTC
We are closely monitoring updates from Microsoft regarding this issue. At this time, they have not provided an estimated timeline for full system restoration. You can follow their updates directly here: https://admin.microsoft.com/#/homepage/:/alerts/MO941162 The impact to Mesh customers is exclusively related to our API integration with Microsoft, meaning remediate functionality is not working.
- monitoring Nov 25, 2024, 10:05 PM UTC
At present all notification queues have cleared and Mesh services are functioning as expected. We're continuing to monitor Microsoft updates here - https://admin.microsoft.com/#/homepage/:/alerts/MO941162
- resolved Nov 26, 2024, 12:21 AM UTC
Although Microsoft report that they are still seeing issues, we have not seen any impact to our service for over 2 hours. We are marking the incident as resolved but will continue to monitor.