SYNAQ experienced a minor incident on September 9, 2019 affecting SYNAQ Cloud Mail, lasting 8h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 09, 2019, 08:35 AM UTC
Dear Clients. SYNAQ Cloud Mail is currently experiencing degraded performance affecting a subset of mailboxes and delays sending and receiving certain mails. Engineers are currently investigating the issue.
- identified Sep 09, 2019, 11:55 AM UTC
SYNAQ engineers have identified the issue and are currently working on a resolution
- resolved Sep 09, 2019, 04:35 PM UTC
SYNAQ Cloud Mail services have been fully restored
- postmortem Sep 13, 2019, 10:25 AM UTC
Summary and Impact to Customers On Monday 9 th September 2019 from 10:00am – 18:35 pm and Tuesday 10 th September 2019 from 08:48am – 11:09 am, SYNAQ Cloud Mail experienced a degraded service incident which caused mail delays and interrupted mailbox use for a subset of clients. The resultant impact of the event was that certain users experienced a delay in mails and their mailboxes were very slow to respond. Root Cause and Solution The root cause of this event was due to a failed controller on a backend storage device. As a result of the failure, all data accessing the storage had to failover to a single data path. This failover caused an abnormal level of usage over the single data path, resulting in increased latency times for data reads and writes, which in turn caused delayed mail delivery and interrupted mail access for a subset of users. In order to resolve this issue, the controller was replaced and dual paths were restored. As a result, mail delivery and access performance was restored to the affected users. Remediation Actions SYNAQ engineers are implementing a storage plan to increase the single data path capacity to handle failover load without impacting users.