Highlight.io incident

Delayed session and error ingest

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on January 21, 2025, lasting 6h 34m. The incident has been resolved; the full update timeline is below.

Started
Jan 21, 2025, 06:53 PM UTC
Resolved
Jan 22, 2025, 01:28 AM UTC
Duration
6h 34m
Detected by Pingoru
Jan 21, 2025, 06:53 PM UTC

Update timeline

  1. investigating Jan 21, 2025, 06:53 PM UTC

    We're investigating a ~2 hour delay in the sessions / errors appearing in the highlight UI.

  2. identified Jan 21, 2025, 11:09 PM UTC

    We have identified ClickHouse async inserts to be the bottleneck in our data sync queue. We are working on a fix.

  3. identified Jan 21, 2025, 11:57 PM UTC

    We've shipped a fix to the underlying issue and are starting to catch up on the data backlog.

  4. monitoring Jan 22, 2025, 12:17 AM UTC

    We're vaught up on all delayed data and are monitoring the current ingest rate.

  5. resolved Jan 22, 2025, 01:28 AM UTC

    The issue is resolved.