Highlight.io incident

Delayed traces ingest

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on February 28, 2025, lasting 12h 17m. The incident has been resolved; the full update timeline is below.

Started
Feb 28, 2025, 03:25 PM UTC
Resolved
Mar 01, 2025, 03:42 AM UTC
Duration
12h 17m
Detected by Pingoru
Feb 28, 2025, 03:25 PM UTC

Update timeline

  1. investigating Feb 28, 2025, 03:25 PM UTC

    Traces are currently delayed 20 minutes. We're investigating.

  2. identified Feb 28, 2025, 03:51 PM UTC

    We've identified the root cause in our ClickHouse database cluster that we're working on addressing.

  3. identified Feb 28, 2025, 04:07 PM UTC

    We're catching up on the trace backlog. ETA 30 minutes.

  4. monitoring Feb 28, 2025, 08:59 PM UTC

    We had another related incident in the trace ingest which created an additional ingest blip. We've tuned the insert concurrency and the backlog is decreasing (ETA 30 minutes).

  5. resolved Mar 01, 2025, 03:42 AM UTC

    All data caught up.