SYNAQ experienced a minor incident on February 13, 2020 affecting SYNAQ Securemail, lasting 4d 18h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 13, 2020, 11:39 AM UTC
Dear Client, SYNAQ Securemail is experiencing intermittent email delays when sending outbound through our platform. Engineers are attending and treating this as a matter of urgency.
- investigating Feb 13, 2020, 12:44 PM UTC
Dear Clients, Our engineers are still investigating the SYNAQ Securemail intermittent outbound email delays. Please be assured that this matter is being treated as a matter of urgency and priority.
- investigating Feb 14, 2020, 09:14 AM UTC
Dear Clients, Our engineers have identified the cause of the intermittent outbound email delays and are working with the upstream provider to resolve this issue as a matter of priority.
- monitoring Feb 17, 2020, 09:48 AM UTC
Dear Clients, The SYNAQ Securemail outbound mail delays are no longer occurring and all mail is flowing optimally. We will continue to monitor the mail flow before a final resolution is sent.
- resolved Feb 18, 2020, 06:28 AM UTC
Dear Clients, The SYNAQ Securemail issue has been resolved and the service has returned to optimal functionality.
- postmortem Mar 02, 2020, 07:56 AM UTC
Summary and Impact to Customers On Thursday 13 th February from 00:14am – Sunday 16 th February 23:14pm, SYNAQ Securemail experienced a service incident which affected the delivery of mail. The impact of the event was an intermittent delay of emails being sent out of the Securemail platform that were then being delivered back inbound to domains using the Securemail platform. Cause and Solution The root cause of this event was due the failure of a core switch, belonging to our hosting provider. The effect of the switch failure caused a variety of different issues within our Securemail environment, resulting in intermittent connection timeouts between the outbound and inbound Securemail servers. This meant that, for some emails, numerous delivery retries were required to ultimately complete delivery. In order to resolve this issue, the failed switch was replaced and SYNAQ engineers worked with Cisco to resolve the timeouts. After many hours of troubleshooting and numerous attempted solutions, the issue was resolved. Whilst we are aware of what initiated this issue, we are continuing to work with our hosting provider and Cisco to ascertain why this switch failure affected the Securemail environment in this way. Actions * We are continuing our investigations with our hosting provider and Cisco to establish why this switch failure affected the Securemail environment in this way. * New monitoring alerts have been built and implemented for early detection of this issue should it recur.