Skylight experienced a major incident on July 22, 2022 affecting Application, lasting 6h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Application
Update timeline
- investigating Jul 22, 2022, 07:39 PM UTC
We are investigating a "clog" in the data processing pipeline that is causing one of the worker servers to be "stuck". This only affect a portion of the customers. If you are affected, you will see missing data from the Skylight dashboard for the last few hours (which is slowly populating). At this stage, we believe the data is safely received in the queue for processing, and once we resolve the issue that causes the "clog" we will be able to backfill the missing data.
- monitoring Jul 22, 2022, 08:34 PM UTC
We have "unclogged" the worker and are "catching up" on the backlog.
- resolved Jul 23, 2022, 02:36 AM UTC
We have completed processing of the backlog.