SYNAQ incident

SYNAQ Branding and Securemail User Interface issue - 15/01/2020

Major Resolved View vendor source →

SYNAQ experienced a major incident on January 15, 2020 affecting SYNAQ Branding and SYNAQ Securemail, lasting 4h 8m. The incident has been resolved; the full update timeline is below.

Started
Jan 15, 2020, 07:21 AM UTC
Resolved
Jan 15, 2020, 11:29 AM UTC
Duration
4h 8m
Detected by Pingoru
Jan 15, 2020, 07:21 AM UTC

Affected components

SYNAQ BrandingSYNAQ Securemail

Update timeline

  1. investigating Jan 15, 2020, 07:21 AM UTC

    Dear Clients, SYNAQ Securemail and branding is currently experiencing an issue where their user interfaces are not available. Engineers are investigating the issue as a matter of urgency.

  2. investigating Jan 15, 2020, 08:04 AM UTC

    Dear Clients, Our engineers are still investigating the SYNAQ Securemail and branding issue. Please be assured that this matter is being treated as a top priority.

  3. identified Jan 15, 2020, 08:22 AM UTC

    Dear Clients, Our engineers have identified the SYNAQ Securemail and branding issue and are working on a resolution. Please take note that Emails are not currently being branded. Email flow is not affected.

  4. monitoring Jan 15, 2020, 09:17 AM UTC

    Dear Clients, Our engineers have implemented a fix for the SYNAQ Securemail and branding issue and are monitoring the service stability. All User interfaces are available and mails are branding. Securemail quarantine backlog busy catching up.

  5. resolved Jan 15, 2020, 11:29 AM UTC

    Dear Clients, The SYNAQ Securemail and branding issue has been resolved and the service has returned to optimal functionality.

  6. postmortem Jan 20, 2020, 09:45 AM UTC

    Summary and Impact to Customers On Wednesday 15th January from 09:21am – 01:29pm, SYNAQ Securemail and Branding experienced a service incident which affected the branding of mail and access of the relevant user interfaces. The resultant impact of the event was that Clients were unable access both the Branding and Securemail user interfaces and mails were not being branded. Root Cause and Solution The root cause of this event was due to a hardware failure in user interface cluster and failover not taking place correctly. As a result, users were unable to access the user interfaces for Branding and Securemail to perform any administrative actions. In conjunction with this all outbound mails were not being branded. In order to resolve this issue, the affected hardware was replaced and all services were restored. There was a delay in information in the Securemail user interface quarantine as the backlog was processed. This backlog took about 3 hours to be fully caught up. Remediation Actions • A project is underway to improve the failover mechanisms should a hardware failover of this magnitude occur again. The work on this project will be completed before the end of January 2020.