mailprotector Outage History

At approximately 1:00 PM ET, we were alerted to slow message processing in Shield. The symptoms were slow message updates and some delivery delays. The team has identified the bottleneck and resolved the issue. The queues are catching up, and performance appears to be normalizing. We will continue to monitor the queue for full resolution.

Email log data stopped shipping to the storage layer. The team has resolved the log shipping, and the logs are recovering. Log backfill will continue well into the afternoon, and we will update this notice once the logs have caught up.

Earlier today, Microsoft identified some of Shield's services as suspicious or compromised. This caused emails to bounce or be delayed. As of approximately 2:30 PM ET, the status began to lift, and we observed a return to expected mail flow. A support request has been opened with Microsoft to help us determine how and why this incident happened. We have not received a response yet. We are not closing this issue as resolved. Our intention is to get to the root of the incident. Our first priority was to find a way to resume normal mail flow, and it appears we have done so. We are continuing to monitor the situation, including the Microsoft notice regarding mail flow issues in Exchange Online, which appear to have been resolved.

We have begun reprocessing and redelivering emails that bounced during the affected time period. Here is what to expect: Most emails should be delivered, but not all of them. Some bounces during the incident may have been legitimate, and those messages will not be recoverable through reprocessing. Duplicates are possible. If a sender resent their email after the incident and the original message is also redelivered through our reprocessing, recipients may receive both. This is expected behavior and not a cause for concern. Reprocessing takes time. We expect this to be completed by midnight ET tonight, though it may finish sooner.

We have been made aware that emails sent from Google Workspace accounts via CloudFilter/SafeSend are failing intermittently. There may also be delays in delivering inbound email due to an issue Google is experiencing. If you have a domain affected by this incident, we recommend temporarily disabling the outbound mail routes for Gmail. You may check the status of Google's incident at https://support.cloud.google.com/portal/system-status?product=WORKSPACE&cssp=true

An update to improve brand-related impersonation attacks was aggressively holding outgoing messages and sending inbound emails to the Junk E-mail folder. The update has been rolled back, and the held outgoing emails have been released. Emails delivered to the Junk E-mail folder must be moved out by users. The aggressive protection was active from approximately 2:00 PM ET to 3:00 PM ET.

Emails that were delivered to the Jail due to the aggressive behavior of the update have been released. The emails may still end up in the Junk E-mail folder if the risk level is medium or low from a new sender.

Microsoft has announced a service issue affecting Exchange Online in M365. We are seeing email delivery delays to M365 domains with temporary error notifications. We will continue to monitor the Microsoft report. Mailprotector's CloudFilter and Shield services are operational.

Microsoft's status has not changed. For more information, please visit https://status.cloud.microsoft/

We will continue to monitor the status Microsoft provides at https://status.cloud.microsoft/. It appears Microsoft is making progress in some of their affected zones. Logs shows some emails are beginning to deliver, but without Microsoft's confirmation of a resolution, we expect deferrals to continue.

Radar is operating as expected. However, AWS has not confirmed that the affected service has been fully resolved. We will continue to monitor the situation.

Connections to CloudMail are operating as expected. We will continue to monitor the status.

Quarantine services are beginning to recover. We continue to monitor the progress reported by AWS and the recovery of queues and services affected by the outage.

A server service was unstable. The team has resolved the issue. Webmail and IMAP access is working as expected. We are continuing to monitor the service.

Inbound delivery is performing as expected. We are continuing to monitor the flow.

The team has resolved email delivery performance, and inbound messages have been arriving as expected since about 1:00 PM ET. However, the issue is still monitored as API-related actions (sender training, message moves, etc.) and logs catch up. Please be aware that emails may sit in a folder a little longer than expected, and Spotlight logs may display an incorrect email location while the database catches up on activity. We expect the issue to be resolved entirely this evening.

We are continuing to monitor.