SYNAQ experienced a critical incident on October 13, 2021 affecting SYNAQ Cloud Mail, lasting 2h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 13, 2021, 10:31 AM UTC
Dear Clients, SYNAQ Cloud Mail is currently experiencing an issue where users are unable to access their mailboxes. Engineers are investigating the issue as a matter of urgency. We will provide an update in 30 minutes
- investigating Oct 13, 2021, 10:58 AM UTC
Dear Clients, SYNAQ Cloud Mail is still experiencing an issue where users are unable to access their mailboxes. Engineers are investigating the issue as a matter of urgency. We will provide an update in 30 minutes
- identified Oct 13, 2021, 11:39 AM UTC
Dear Clients, Engineers have identified the Cloud Mail issue and are working on the root cause as a matter of urgency. Users are currently able to access their mailboxes. We will provide a further update in 30 minutes
- monitoring Oct 13, 2021, 12:06 PM UTC
Dear Clients, The SYNAQ Cloud Mail issue has recovered. Our engineers are monitoring the service stability. We will provide a further update in 30 minutes
- resolved Oct 13, 2021, 12:38 PM UTC
Dear Clients, The SYNAQ Cloud Mail issue has been resolved and the service has returned to optimal functionality.
- postmortem Oct 20, 2021, 11:44 AM UTC
Summary and Impact to Customers On Wednesday 13th October 2021 from 12:03 to 14:06, SYNAQ Cloud Mail experienced a mail authentication incident. The resultant impact of the event was that users received an authentication pop-up message when trying to login and experienced slow access to mail. Root Cause and Solution The root cause of this event was due to an abnormal amount of modification requests being received from the SYNAQ API. The increased writes and load on the master LDAP server prevented the LDAP replicas from being able to replicate the LDAP changes from the master. Since LDAP replicas are responsible for servicing requests for authentication and mail delivery, and were unable to sync correctly from the master LDAP server in this instance, authentication and mail delivery requests were not processed. To resolve this issue, the primary server and all replicas had to be restarted to re-establish the syncing between LDAP servers. Once all the LDAP servers were in sync, authentication requests were once again being processed and users could access their mail. Remediation Actions • More stringent limits on the API are to be put in place to limit the number of requests that can come in from a single IP address to prevent increased load on the LDAP servers. • The SYNAQ infrastructure teams are working in collaboration with the 3rd Party vendor to improve and harden our LDAP configuration.