Highlight.io incident

Delayed traces ingest.

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on May 11, 2024, lasting 16h 55m. The incident has been resolved; the full update timeline is below.

Started
May 11, 2024, 01:10 AM UTC
Resolved
May 11, 2024, 06:06 PM UTC
Duration
16h 55m
Detected by Pingoru
May 11, 2024, 01:10 AM UTC

Update timeline

  1. investigating May 11, 2024, 01:10 AM UTC

    We're investigating a delay in ingesting traces due to an OOM error in our MSK collector. Current delay is ~30 minutes.

  2. monitoring May 11, 2024, 02:11 AM UTC

    Root cause resolved; catching up on the backlog now.

  3. resolved May 11, 2024, 06:06 PM UTC

    Backlog resolved as of May 10 19:58 PT