SYNAQ experienced a major incident on March 4, 2020 affecting SYNAQ Cloud Mail, lasting 2h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 04, 2020, 09:01 AM UTC
Dear Clients, SYNAQ Cloud Mail is experiencing an issue where a subset of users cannot access their mailbox. Engineers are investigating this as a matter of urgency.
- identified Mar 04, 2020, 10:07 AM UTC
Dear Clients, Our engineers have identified the SYNAQ Cloud Mail issue and are working on the resolution as a matter of priority.
- resolved Mar 04, 2020, 11:05 AM UTC
Dear Clients, The Cloud Mail issue, affecting a subset of clients, has been partially resolved. Affected clients can now access their mailboxes. Engineers will continue to work on mailboxes to restore any mail that may be missing for the last few hours.
- postmortem Mar 11, 2020, 08:18 AM UTC
Summary and Impact to Customers On Wednesday 4th March 2020 from 11:01 to 13:05, SYNAQ Cloud Mail experienced a mail authentication incident. The resultant impact of the event was that a subset of users could not access the platform. Root cause and Solution The root cause of this event was due to a corruption of certain files in the Cloud Mail stores MYSQL databases. This caused the services on that particular store to stop and then users on that particular store were unable to authenticate and access their mailboxes. In order to resolve this issue, we restored the database from backups which allowed the services to be restarted and users to once again access their mailboxes. Remediation Actions • A call has been logged with our 3rd party provider to do some analysis of the MYSQL database information to and try and determine why this corruption occurred and what can be done in future to mitigate this. • A new monitoring alert has been built to try and detect early any corruptions of this nature.