SYNAQ experienced a minor incident on July 31, 2023 affecting SYNAQ Cloud Mail, lasting 1h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 31, 2023, 07:01 AM UTC
Dear SYNAQ Client, We have identified a minor Cloud Mail incident where a subset of clients are experiencing slow email responses on webmail. Engineers are investigating as a matter of urgency, and we will continue to provide updates as they become available. We apologise for the inconvenience caused. Sincerely, SYNAQ Technical Team
- investigating Jul 31, 2023, 07:02 AM UTC
We are continuing to investigate this issue.
- investigating Jul 31, 2023, 07:31 AM UTC
Dear SYNAQ Client Our engineers are still investigating as a matter of urgency and will continue to provide updates as they become available. We apologise again for the inconvenience caused. Sincerely, SYNAQ Technical Team
- resolved Jul 31, 2023, 08:04 AM UTC
Dear SYNAQ Client, Our engineers have both identified and resolved the SYNAQ Cloud Mail incident. The service has returned to optimal functionality and we will continue to monitor the service closely. Please ensure to log out and back into your account to resume services. Thank you for your patience and understanding on this matter. Sincerely, SYNAQ Technical Team
- postmortem Jul 31, 2023, 12:26 PM UTC
On Monday, 31 July 2023, 08:25, Cloud Mail experienced a slow response on SYNAQ Webmail platform, with incoming and outgoing mail. The following concerns were raised. * Slow response on incoming and outgoing mail, and; * Server error when accessing Webmail. The resultant impact of the event was that a subset of users who were using Webmail experienced a slow response on incoming and outgoing mail. The root cause of this event was due to degraded memory performance across the Cloud Mail Proxy stack. This resulted in slow connections on email when trying to login via Webmail. To resolve this incident, our engineers removed one proxy at a time from the cluster, restarted the relevant proxy services on it and added it back to the cluster. This immediately resolved the degraded state and restored the platform to normal operations. Upon investigation of the root cause of the incident the team at SYNAQ will implement the below remedial actions: · Updating the existing monitoring for memory swapping to perform remediation actions at a much earlier time.