Highlight.io incident

Session / Errors Ingest Delay

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on October 8, 2024, lasting 5h 16m. The incident has been resolved; the full update timeline is below.

Started
Oct 08, 2024, 04:53 PM UTC
Resolved
Oct 08, 2024, 10:10 PM UTC
Duration
5h 16m
Detected by Pingoru
Oct 08, 2024, 04:53 PM UTC

Update timeline

  1. investigating Oct 08, 2024, 04:53 PM UTC

    We're currently experiencing a data ingest delay of approximately 1 hour for sessions and errors.

  2. identified Oct 08, 2024, 07:09 PM UTC

    We're identified the issue in a performance regression in our error grouping code.

  3. identified Oct 08, 2024, 07:09 PM UTC

    We have a fix in place that is causing us to catch up on the backlog of data. Current backlog is roughly 1 hour, with an ETA to catch up of 5pm PT.

  4. monitoring Oct 08, 2024, 08:50 PM UTC

    The root cause is resolved. We are catching up on the backlog of data with data now delayed < 30 minutes.

  5. resolved Oct 08, 2024, 10:10 PM UTC

    We've caught up on the data backlog.