Skylight experienced a major incident on April 7, 2017, lasting 6h 30m. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Apr 07, 2017, 01:56 AM UTC
We are performing a major upgrade on the data processing pipeline. This has taken longer than expected so far, and as our result we are not processing data at this time. Data processing will resume once we complete the upgrade.
- identified Apr 07, 2017, 03:15 AM UTC
We have identified the issue that is preventing our data processing workers to make progress after the migration. We are currently working on addressing the problem.
- monitoring Apr 07, 2017, 07:44 AM UTC
We have resolved the issue that was blocking our data processing pipeline from making progress. We are currently re-processing all of the data we received during the maintenance window. Unfortunately, we need to prune some corrupted data, which will cause a small amount of data about individual requests to be missing between 18:00 and 20:00 PDT.
- resolved Apr 07, 2017, 08:27 AM UTC
We have completed the system upgrade and our data processing is fully caught up.