SYNAQ incident
SYNAQ Cloud Mail - Intermittent SMTP (Outgoing Mail) Sending Failures
SYNAQ experienced a minor incident on July 6, 2023 affecting SYNAQ Cloud Mail, lasting 3h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 06, 2023, 04:44 PM UTC
Dear SYNAQ Client, We have identified a minor Cloud Mail incident where a subset of clients are experiencing intermittent failures on Sending Outgoing Mails. (Mails stuck in outbox, mail sending failures, cannot upload attachments and send). Engineers are investigating as a matter of urgency, and we will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
- investigating Jul 06, 2023, 05:37 PM UTC
Dear SYNAQ Client, Our engineers are still investigating as a matter of urgency and will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
- investigating Jul 06, 2023, 06:37 PM UTC
Dear SYNAQ Client, Our engineers are working together with our upstream providers to further investigate the incident. We will continue to provide updates as they become available. We apologise for the inconvenience caused and would like to assure you that this is being treated as a matter of importance and urgency. Sincerely, SYNAQ Technical Team
- resolved Jul 06, 2023, 07:45 PM UTC
Dear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
- postmortem Jul 12, 2023, 02:53 PM UTC
On Thursday, 6 July 2023, 15:25, Cloud Mail experienced intermittent sending failures on the SYNAQ Webmail platform, with outgoing mail. The following concerns were raised: * Slow response when sending email which contains attachments, and; * Emails stuck in Outbox. The resultant impact of the event was that a subset of users who were using Zimbra connector for Outlook were experiencing either a slow response on webmail or ActiveSync and/or timeout errors when sending emails containing large attachments. The root cause of this event was due to a degraded connection state between the load balancer and the mail cluster. This resulted in slow connections on email when trying to login via HTTPS, POP3/S, IMAP/S, SMTP/S, Zimbra connector for Outlook. To resolve this incident, we moved all sessions to the secondary load balancer. This immediately resolved the degraded state and restored the platform to optimal operations. Upon investigation of the root cause of the incident, the SYNAQ team will implement the below remedial actions: · New Monitoring to be implemented to detect any failed states on the Load-balancer, and; · Further streamlining the internal incident management process.