Highlight.io incident

Degraded data ingest

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on February 2, 2024, lasting 10h 34m. The incident has been resolved; the full update timeline is below.

Started
Feb 02, 2024, 10:21 AM UTC
Resolved
Feb 02, 2024, 08:56 PM UTC
Duration
10h 34m
Detected by Pingoru
Feb 02, 2024, 10:21 AM UTC

Update timeline

  1. investigating Feb 02, 2024, 10:21 AM UTC

    We are noticing a data delay as of 11:45pm PST.

  2. identified Feb 02, 2024, 10:34 AM UTC

    We've identified an unintentional AWS SDK upgrade that was rolled out that is causing the ingest workers to crash.

  3. monitoring Feb 02, 2024, 12:19 PM UTC

    We are catching up on the data backlog now. Data is still about 6 hours delayed.

  4. resolved Feb 02, 2024, 08:56 PM UTC

    We have caught up on all delayed data.