mailprotector Outage History

This morning, a third-party package repository used by our virus scanning infrastructure experienced an outage following an attack on its systems. Because our virus scanning instances rely on this repository at startup to pull the latest files and configurations, affected instances were unable to start. This reduced the capacity of our virus-scanning cluster below demand. When a virus scan cannot complete, our system quarantines the email as a precaution. As a result, emails processed during this window were quarantined rather than delivered to inboxes. No emails were lost. Affected emails can be released from quarantine by users or admins. Resolution The virus scanning cluster has been reconfigured to pull packages directly from the source repository, eliminating the dependency on the third-party intermediary. Monitoring and alerting thresholds have also been updated to surface this class of issue more quickly in the future. The issue is fully resolved.

This incident has been resolved.

An update released at approximately 2:00 PM ET today was designed to improve protection against brand impersonation. It behaved more aggressively than intended, holding and jailing legitimate emails, particularly those with signatures containing social media links. The update was rolled back at approximately 2:50 PM ET. This issue is resolved. Actions taken to remediate after rollback: - Inbound emails delivered to Junk have been reprocessed to ensure appropriate risk and moved to the Inbox. - Released jailed emails; they may still land in Junk if the sender is new or carries a medium risk level. - Outgoing emails that were held have been released. No action needed.

This incident has been resolved.

The log ingestion issue is resolved, and logs have been backfilled.

The fix has been deployed. New emails are being placed in the appropriate location as expected per X-ray log details. Emails that were not moved prior to the fix will remain in their current location. Moving the message will create an opportunity for Shield to act on the message, if needed.

This incident has been resolved.

This incident has been resolved.

Quarantine queues are continuing recovery and services appear to be operating as expected. We will monitor the AWS status through the morning.

This incident has been resolved.

A permissions change caused the Mailprotector Console to lose connection to a status database. The result was a false report of mail flow errors in the Console and email notifications to admin accounts configured for the alerts. The permissions have been corrected, and the mail flow status in the Console for CloudFilter shows accurate information. There was no mail flow interruption.

Inbound delivery has continued to perform as expected. This incident is resolved.

A modification to handling ARC seal authentication was deployed late yesterday evening to handle an outlier situation observed over the past few months. The modification created a mail loop in specific messages that were missing some ARC seal authentication information in the headers. This specific situation caused the emails to look up the MX record for delivery, and those domains with a Shield MX record caused the email to loop. A fix for this situation was deployed shortly after the mail loop was detected. The fix now accounts for the rare circumstances involving the ARC seal and will gracefully handle mail delivery.

Email delivery and API activity are operating as expected.

User sync errors affecting Google Workspace customers have been resolved. If an error still exists for a domain, please delete and recreate the user sync in the Mailprotector Console or open a request with the support team for guidance.

Since April 29, there have been unnoticeable bounces. While we cannot be certain that Microsoft will not repeat the issue, the status is considered resolved at this time. If you receive a bounce for "banned sending IP" or "blocked using Spamhaus", please open a support request.

Over the weekend, a small pool of inbound gateway servers bounced incoming emails to CloudFilter because of a database connection issue. The NDR message indicates: The response from your-domain-com.inbound.emailservice.io was: 5.7.1 [email protected]: Recipient address rejected: Access denied The team responded and resolved the issue quickly. Fortunately, weekends have a comparatively low email volume. However, if a user was expecting an email and did not receive it, they may need to request the sender to retry sending it. A thorough investigation of the incident is being performed, and the issue will be adjusted, monitored, and mitigated for the future.

Monitoring systems alerted the team to an issue with approximately 20% of inbound email processing. The issue has been identified and resolved. The issue will have impacted emails that show a delay in the logs this morning, but no further impact is expected.

Earlier this morning, Bracket encryption scaling added resources that did not respond to connection requests correctly. Users may have received a bounce notification if their encrypted email attempt connected to this subset of resources. The bounce notification error was: The bounce is: : host encrypt.bracket.email[18.223.92.55] said: 550 A secure TLS connection is required. The problematic resources were identified and removed, and the issue has been resolved. Users can send encrypted Bracket emails as expected.

Mail flow is functioning at expected performance levels.

Inbound mail flow from CloudFilter is operating as expected. The issue has been resolved.

Deferrals of emails have slowed substantially but have not ceased altogether. Deferrals to certain email service providers will always occur due to several factors, including domain and sender reputations. Mailprotector deems the IP reputation problem resolved at this time. Mailprotector does not control domain and sender reputations. IP reputation management is influenced by Mailprotector's sending infrastructure management, but it is not solely in our control. Deferrals may happen on occasion. Users will see a deferral notice under the right circumstances as we continue to manage the sending infrastructure to build additional resilience and strictly enforce the sending policy. If a deferral or bounce is questionable, please open a support request, and Partner Success will gladly explain the situation as best as possible. In a pinch, the option is to send emails directly from the email service provider rather than CloudFilter's smarthost relay.

We have confirmed the TLS connections between the affected CloudFilter outbound gateway servers and Bracket have been working as expected.

An issue that started at 12:21 PM ET resulted in lost logs from 12:20 PM to 2 PM and likely duplicated some logs in the early morning. AWS performed a maintenance task that restarted a transport server in an ungraceful manner. This caused the logging service to lose its place in the log that it was duplicating. The logging service started at the beginning of the log file and writing logs. The problem was discovered, and it forced a rotation of the mail log (to start over fresh) and a re-addition of the affected server to the load balancer. The event may have duplicated some log messages, dropped others, and created odd-looking data in the mail flow logs in the Mailprotector Console during the event. However, mail flow was not affected by this event.