Resolved after 8 hours 1 minute, affecting SYNAQ Cloud Mail, on June 11, 2019.
Summary and Impact to Customers On Monday 10th June from 1:38pm to Tuesday 11th June 4:55pm, SYNAQ Cloud Mail experienced a minor service incident which caused mail delays for a subset of clients. The resultant impact of the event was that certain users experienced mail delays of up to 2 hours for some of their mail. Root Cause and Solution The root cause of this event was due to a failed controller on a backend storage device. As a result of the failure, all data accessing the storage had to failover to a single data path. This failover caused an abnormal level of usage over the single data path, resulting in increased latency times for data reads and writes, which in turn caused mail delivery to users mailboxes to be delayed. In order to resolve this issue, the controller was replaced and dual paths were restored. As a result, mail delivery performance was restored to the affected users. Remediation Actions • Whilst the built-in redundancy on the storage array worked to prevent a complete loss of access to mailbox data, SYNAQ engineers are working with our storage vendor to increase the single data path capacity to handle failover load without impacting users.
Affected components
Update timeline
- investigating Jun 11, 2019, 06:53 AM UTC
Dear Clients, SYNAQ Cloud Mail is experiencing mail delays. We are aware of the issue and we are working on the resolution as a matter of urgency.
- monitoring Jun 11, 2019, 01:53 PM UTC
Dear Clients, The Cloud Mail incident has been resolved. Mail queues are currently working through the backlog and engineers are monitoring
- resolved Jun 11, 2019, 02:55 PM UTC
Dear Clients Mail flow is optimal again
- postmortem Jul 23, 2019, 02:01 PM UTC
Summary and Impact to Customers On Monday 10th June from 1:38pm to Tuesday 11th June 4:55pm, SYNAQ Cloud Mail experienced a minor service incident which caused mail delays for a subset of clients. The resultant impact of the event was that certain users experienced mail delays of up to 2 hours for some of their mail. Root Cause and Solution The root cause of this event was due to a failed controller on a backend storage device. As a result of the failure, all data accessing the storage had to failover to a single data path. This failover caused an abnormal level of usage over the single data path, resulting in increased latency times for data reads and writes, which in turn caused mail delivery to users mailboxes to be delayed. In order to resolve this issue, the controller was replaced and dual paths were restored. As a result, mail delivery performance was restored to the affected users. Remediation Actions • Whilst the built-in redundancy on the storage array worked to prevent a complete loss of access to mailbox data, SYNAQ engineers are working with our storage vendor to increase the single data path capacity to handle failover load without impacting users.
Your product runs on more than SYNAQ
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services