Highlight.io incident

Delayed session and error ingest

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on August 8, 2024, lasting 44m. The incident has been resolved; the full update timeline is below.

Started
Aug 08, 2024, 06:38 PM UTC
Resolved
Aug 08, 2024, 07:22 PM UTC
Duration
44m
Detected by Pingoru
Aug 08, 2024, 06:38 PM UTC

Update timeline

  1. investigating Aug 08, 2024, 06:38 PM UTC

    As of 10:32 am PT we're experiencing a delay in ingesting sessions and errors.

  2. identified Aug 08, 2024, 06:51 PM UTC

    We've identified an issue with our error group embeddings processing that has been causing a backlog.

  3. identified Aug 08, 2024, 06:52 PM UTC

    We've scaled our LLM embeddings endpoint and are catching up on the backlog. ETA to resolution <1 hour.

  4. resolved Aug 08, 2024, 07:22 PM UTC

    We have now fully caught up on the data backlog.