mailprotector incident

Inconsistent inbound logs through CloudFilter

Notice Resolved View vendor source →

mailprotector experienced a notice incident on October 17, 2024, lasting —. The incident has been resolved; the full update timeline is below.

Started
Oct 17, 2024, 07:39 PM UTC
Resolved
Oct 17, 2024, 04:30 PM UTC
Duration
—
Detected by Pingoru
Oct 17, 2024, 07:39 PM UTC

Update timeline

  1. resolved Oct 17, 2024, 07:39 PM UTC

    An issue that started at 12:21 PM ET resulted in lost logs from 12:20 PM to 2 PM and likely duplicated some logs in the early morning. AWS performed a maintenance task that restarted a transport server in an ungraceful manner. This caused the logging service to lose its place in the log that it was duplicating. The logging service started at the beginning of the log file and writing logs. The problem was discovered, and it forced a rotation of the mail log (to start over fresh) and a re-addition of the affected server to the load balancer. The event may have duplicated some log messages, dropped others, and created odd-looking data in the mail flow logs in the Mailprotector Console during the event. However, mail flow was not affected by this event.