Highlight.io incident

Delayed session / error ingest

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on April 25, 2024, lasting 8h 8m. The incident has been resolved; the full update timeline is below.

Started
Apr 25, 2024, 03:16 PM UTC
Resolved
Apr 25, 2024, 11:24 PM UTC
Duration
8h 8m
Detected by Pingoru
Apr 25, 2024, 03:16 PM UTC

Update timeline

  1. investigating Apr 25, 2024, 03:16 PM UTC

    We're seeing a delay in sessions / errors of about an hour. Working to establish the root cause.

  2. identified Apr 25, 2024, 03:35 PM UTC

    We're found the root cause to be a temporary misconfiguration in our main queue worker leading to OOMs. We've deployed a fix and are catching up on the backlog.

  3. resolved Apr 25, 2024, 11:24 PM UTC

    Ingestion backlog resolved as of 11:45am PT.