Skylight incident

Data Processing Outage

Critical Resolved View vendor source →

Skylight experienced a critical incident on May 15, 2020 affecting Application, lasting 10h 4m. The incident has been resolved; the full update timeline is below.

Started
May 15, 2020, 07:16 AM UTC
Resolved
May 15, 2020, 05:20 PM UTC
Duration
10h 4m
Detected by Pingoru
May 15, 2020, 07:16 AM UTC

Affected components

Application

Update timeline

  1. investigating May 15, 2020, 07:16 AM UTC

    We are experiencing a major network outage on AWS affecting most of our EC2 instances. Due to the scale of this outage, we are expecting connectivity issues from agent and potential loss of report data submitted during this time.

  2. identified May 15, 2020, 08:51 AM UTC

    The network issue has been resolved. We have determined that the data sent during the outage period is not recoverable, and have proceeded to unblock processing new data instead.

  3. monitoring May 15, 2020, 09:08 AM UTC

    We have resumed processing data on all workers.

  4. resolved May 15, 2020, 05:20 PM UTC

    This incident has been resolved.