Mesh Security had 24 outages in the last 2 years totaling 91h 10m of downtime — averaging 1 incident per month.
There were 24 Mesh Security outages since June 20, 2024 totaling 91h 10m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating short email delays. We will post an update as soon as possible.
Queue times have returned to normal. Thank you for your patience.
We are currently investigating an issue impacting our detection engine which is resulting in some emails incorrectly receiving a phishing verdict.
A fix has been implemented and we are monitoring.
The issue has been fixed as of 3pm Eastern Time. Unfortunately, we do not have a mechanism to automatically release those emails that were incorrectly given a phishing verdict. Thank you for your patience while we worked through this.
We are currently investigating some inbound delivery queues on our US infrastructure. We will provide an update as soon as possible.
Queue volume is dropping steadily and the infrastructure team is monitoring performance.
Inbound queues have cleared and delivery times have returned to normal. Thank you for your patience.
We are currently investigating this issue.
We have identified the issue and a fix has been implemented.
A fix has been implemented and we are monitoring the results.
Queue times have returned to normal. We're continuing to monitor.
We are currently investigating some inbound delivery queues on our EU infrastructure.
The team has identified the root cause and is working on a fix.
Message processing has returned to normal and all queues have cleared. We will continue to monitor.
The team has identified an issue causing some inbound email delivery delays. A fix is being deployed shortly.
The issue has been resolved and email delivery has returned to normal.
Our infrastructure team has identified some email processing delays on our US service.
Queues have returned to normal and we are monitoring system stability.
We are marking this incident as resolved but will continue to monitor closely.
We are currently investigating an issue where emails are slow to process on Mesh 365 on our US service. Mesh Gateway and Mesh Unified is not impacted.
The issue has been addressed and mails are processing as normal on Mesh 365. We will continue to monitor. Thank you for your patience while we worked on this.
Spoke too soon. Apologies for the miscommunication. Outbound processing queues have returned to normal. There is still a heavy queue Inbound on the US infrastructure with delivery into 365 static. Microsoft are reporting that all services are operational we will continue to monitor ...
We expect all messages to be delivered in ~30 mins. New emails are being delivered as normal.
Mail delivery is functioning as expected. We're continuing to monitor the situation. We're closing out this status.
We're observing slow mail processing on the US infrastructure.
Mail processing queues have returned to normal. We're continuing to monitor our infrastructure. For now everything is operational. There is a maintenance window in Azure this evening which we believe will resolve the the underlying throughput issue within their Networks.
We're observing a recurrence of the issue on Azure whereby emails are slow to process on the US infrastructure. We're currently monitoring the situation and have reached out to Microsoft for a status update.
Mail processing queues have returned to normal. We're continuing to monitor our infrastructure. For now everything is operational.
Microsoft is currently experiencing issues delivering emails to tenants within the EU. This is NOT a Mesh issue and this status is informational. Impacted emails will be temporarily deferred and we will continue to retry delivery. Microsoft Status: "We're investigating an issue in Europe where some users may experience delays when sending and receiving email messages. For more details, please look for EX1144276 in the admin center."
Our infrastructure team has been monitoring the situation. Inbound/outbound emails are being delivered more reliably and email queues are reducing in size. Latest update from Microsoft (EX1144276): "The new throttling rule has successfully been deployed. We’ve received confirmation from some of the previously affected users that they can now send and receive email. Additionally, our telemetry shows that the new throttling rule is having the desired effect and providing relief."
Email traffic has returned to normal and Microsoft has marked the incident as resolved. Microsoft Status: "After monitoring the service and receiving confirmation from previously affected users, we’ve confirmed that impact has been remediated. The service will now resume periodic attempts to deliver emails that were sent during the impacted period."
We are aware that one of our outbound IP addresses was listed on Spamhaus' RBL. This IP has been removed from our sending relays and the team is actively working with Spamhaus to identify a root cause. At this time, we do not anticipate any ongoing impact to email delivery and we are closely monitoring for further reports. No action is required from end customers or partners.
Our team has received further feedback from Spamhaus earlier this morning. The listings were caused by an end user emailing a spamtrap/honeypot operated by Spamhaus. The honeypot domain is a typosquat of a well known brand and the emails sent were not malicious traffic. We have reached out to the MSP so they can advise their end user of the mistake/typo.
The issue has been identified and a fix is being implemented.
A fix has been implemented and mail flow has been restored. Delivery Event logging is still behind. We're working to restore that.
This incident has been resolved.
We are currently investigating an issue with inbound mail flow in the EU.
HUB access has been restored. Mails should begin flowing shortly.
A fix has been implemented and we're currently monitoring.
We are continuing to monitor for any further issues.
All systems are operational.
There appears to be some Email Delays on the EU Inbound Gateway Infrastructure. We're currently investigating.
We're currently monitoring inbound delivery on the EU. Delivery times are currently within a normal range.
We believe the issue is localised to Microsoft O365s AUS Infrastructure, as all other inbound traffic on the Mesh Gateway Service is unaffected. We'll continue to monitor.
Microsoft have acknowledged the issue affecting Exchange Online in Australia. https://x.com/MSFT365Status/status/1925443356118688152 We'll continue to monitor.
All Mesh services are continuing to function normally, however, we are still seeing mail flow irregularities to Australian customers on Microsoft 365. This is not an issue specific to customers using Mesh. Microsoft are now reporting that they have identified the cause and have put in a fix - https://x.com/MSFT365Status/status/1925482680646357013 We will continue to monitor.
We can confirm successful delivery of emails to customers based in Australia on Microsoft 365. This has now been resolved and we will continue to monitor. Thank you.
The team has identified a spike in email traffic on our EU infrastructure. Email delivery may be delayed by a short period but systems are auto scaling and queues are clearing steadily
Email traffic has returned to normal and all queues have cleared.
We are investigating the slow performance of the portal on our EU service. Email processing is not impacted.
The team have made some changes to mitigate the performance issues experienced with the LET
We are currently investigating
The issue has been identified and a fix is being implemented.
The incident has been resolved and email processing times have returned to normal. Thank you for your patience while we worked through this issue.
We are currently investigating this issue.
The issue has been identified and impacts a small subset of email which is still in a queue. We are working to resolve this issue. New email is being processed as normal.
The small queue of emails has now cleared. All emails are processing as normal. Thank you for your patience while we worked on this issue.
We are investigating reports of emails being deferred inbound on our US service
We have identified the issue. Microsoft is rate limiting our delivery of emails inbound on our US infrastructure. Our engineers are working on a fix. Any deferred emails will be delivered once the issue is fixed.
Our engineering team have added additional delivery ranges to mitigate the issue. Some emails are being delivered while others are being deferred. Any deferred emails will be delivered once a fix has been implemented. We are working with Microsoft on this. We will provide a further update in due course. Thank you for your patience.
We have deployed our failover delivery range which has seen an improvement, but Microsoft are still rate limiting delivery. We are continuing to work with them to fix the issue.
While our team are continuing to rotate our delivery ranges to ensure the maximum volume of email can be delivered, Microsoft continue to rate limit email delivery from our ranges. We are continuing to work with their support teams to fix this. Thank you again for your patience.
A fix has now been implemented. Microsoft has stopped rate limiting our delivery ranges. It will take a little bit of time for the queues to clear. We will post another update shortly.
All queues have now cleared. We will continue to monitor. Thank you for your patience.
The team has identified the issue and queues are clearing. Some mail flow is delayed.
Queues have cleared and processing times have returned to normal.
The team is investigating some intermittent processing delays for inbound email on our US service. We will provide a further update shortly.
Email is now flowing as expected. Our team identified that the delay was caused by an ongoing Azure issue which impacted one of our filtering components. Our server redundancy handled the increased load. We're continuing to monitor processing times in case further action is needed.
While Azure is still continuing to report an issue, this is not impacting our service. We're updating our incident status to resolved but will continue to monitor.