mailprotector experienced a minor incident on October 1, 2024 affecting Shield, lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 01, 2024, 01:45 PM UTC
Email delivery delays have been observed on Shield, and the team is investigating the issue. At this time, the delays are over 15 minutes.
- investigating Oct 01, 2024, 02:14 PM UTC
Some emails may be delayed over 30 minutes. The team is investigating the issue.
- identified Oct 01, 2024, 05:59 PM UTC
The team has identified the issue, and mail flow performance has improved. Email delivery rate is just under one minute and is continuing to improve.
- monitoring Oct 01, 2024, 06:51 PM UTC
Mail flow has been restored to regular operation. The team is continuing to monitor the situation and ensure a complete resolution.
- resolved Oct 02, 2024, 02:52 PM UTC
Mail flow has remained healthy, and email delivery performance is resolved.
- postmortem Oct 02, 2024, 02:52 PM UTC
An organization sent a poorly constructed email just before 8:00 AM ET on Tue, Oct 2, with over 2,000 email addresses in the CC field. The header of the email was too long to process as expected, creating a unique situation that contributed to a large amount of resource utilization. Once the team identified the email message, custom handling for this poorly formed email was built. This allowed the message to be processed as expected without increasing resources. Future incidents with poorly constructed emails that exceed typical header data will be handled gracefully.