Skylight incident

Data processing delay

Minor Resolved View vendor source →

Skylight experienced a minor incident on November 28, 2018 affecting Application, lasting 2d 14h. The incident has been resolved; the full update timeline is below.

Started
Nov 28, 2018, 10:37 AM UTC
Resolved
Dec 01, 2018, 12:39 AM UTC
Duration
2d 14h
Detected by Pingoru
Nov 28, 2018, 10:37 AM UTC

Affected components

Application

Update timeline

  1. investigating Nov 28, 2018, 10:37 AM UTC

    We are currently investigating this issue.

  2. identified Nov 29, 2018, 12:17 AM UTC

    We've identified the issue and have a hotfix partially deployed. Data processing has resumed for some applications and we anticipate that it will resume for others soon. At this point, we do not believe that any data has been lost.

  3. monitoring Nov 29, 2018, 01:41 AM UTC

    We've deployed the hotfix to all of our workers and are processing data for all applications. It may take some time before processing is caught up, but we do not anticipate data loss. We continue to work towards a more permanent resolution to the issue.

  4. monitoring Nov 29, 2018, 03:58 PM UTC

    Our workers continue to process data for all applications. We still anticipate a full recovery without data loss.

  5. monitoring Nov 29, 2018, 05:13 PM UTC

    Note, the uptick "Data Processing Delay" is due to raising the maximum value displayed. Previously, 360 was the highest value we would show. We are now displaying the true value.

  6. monitoring Nov 29, 2018, 08:44 PM UTC

    We are continuing to progress and slowly catch up with the backlog. In the meantime, we're investigating ways to process the data more quickly.

  7. monitoring Nov 30, 2018, 01:20 AM UTC

    Data processing has caught up for some applications. We continue to work on the backlog for the remaining applications.

  8. monitoring Nov 30, 2018, 09:14 AM UTC

    We have finished processing the backlog for the majority of our customers. We expect to be fully caught up in the next few hours and return to the normal data processing capacity for Friday daytime (U.S.) traffic.

  9. monitoring Nov 30, 2018, 03:15 PM UTC

    The backlog has been resolved for most applications. We anticipate that the remaining processing will be resolved within the next day.

  10. monitoring Nov 30, 2018, 06:42 PM UTC

    We are continuing to monitor for any further issues.

  11. monitoring Nov 30, 2018, 09:02 PM UTC

    We've deployed additional resources to resolve the final backlog.

  12. resolved Dec 01, 2018, 12:39 AM UTC

    We have finished processing the backlog and returned to normal data processing capacity.

  13. postmortem Dec 05, 2018, 05:59 AM UTC

    We have [published](https://blog.skylight.io/november-outage-post-mortem/) a post-mortem for this incident on our blog.