mailprotector experienced a minor incident on January 29, 2024 affecting CloudFilter and SafeSend Email Security, lasting 3h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 29, 2024, 08:32 PM UTC
The team is currently investigating inbound mail delays. Delayed mail is primarily associated with a singular instance, and the instance has been temporarily removed from production. This should improve delivery times while the team continues to track down the root of the issue.
- monitoring Jan 29, 2024, 09:56 PM UTC
Maintenance was performed on the impacted instance. It has been added back into production and we are no longer seeing delivery delays. We will continue to monitor performance.
- resolved Jan 29, 2024, 11:57 PM UTC
One of the inbound transport servers was automatically rebooted for maintenance and could not handle the imposed load as it scaled back up. A server reboot is typically infrequent and mundane, but this particular transport server did not respond as typically expected. While the other servers could handle their current load, it was decided to resize in order to handle unexpected periods of higher demand in the future. All inbound transport servers have been resized to prevent similar delays, and more monitoring/alerting metrics have been added.