SynchroNet incident

Users receiving "Not Authorized" screen upon login

Critical Resolved View vendor source →

SynchroNet experienced a critical incident on October 11, 2020 affecting API, lasting 47m. The incident has been resolved; the full update timeline is below.

Started
Oct 11, 2020, 10:22 AM UTC
Resolved
Oct 11, 2020, 11:09 AM UTC
Duration
47m
Detected by Pingoru
Oct 11, 2020, 10:22 AM UTC

Affected components

API

Update timeline

  1. investigating Oct 11, 2020, 10:22 AM UTC

    We are currently investigating this issue.

  2. identified Oct 11, 2020, 10:30 AM UTC

    We have identified the cause of the issue and are currently implementing a fix. One of our database servers is experiencing a storage issue. Modifications are in progress to resolve this issue.

  3. monitoring Oct 11, 2020, 10:39 AM UTC

    We have completed the necessary modifications to the database server. Initial checks indicate functionality has been restored.

  4. resolved Oct 11, 2020, 11:09 AM UTC

    Multiple customers have confirmed that access to CLICK has been restored. All logs and metrics have returned to normal.

  5. postmortem Oct 11, 2020, 03:13 PM UTC

    At SynchroNet, we are committed to providing quality and value to our customers. We take any disruption of our services very seriously, and sincerely apologize for any inconvenience this incident may have caused. Thankfully this issue was quick to resolve, but it surfaced an area of our infrastructure that we do not currently include in our proactive monitoring of CLICK’s overall health. We have conducted an internal analysis of how we can prevent similar issues in the future, and will be implementing additional measures to surface potential issues before they can cause an incident. We value every one of our customers, and look forward to continuing to provide you with the best products and services we can, every day. ‌ Adam Platt, Technical Product Manager