Highlight.io incident

Sessions / Errors ingest delay

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on April 29, 2024, lasting 4h 33m. The incident has been resolved; the full update timeline is below.

Started
Apr 29, 2024, 08:02 PM UTC
Resolved
Apr 30, 2024, 12:35 AM UTC
Duration
4h 33m
Detected by Pingoru
Apr 29, 2024, 08:02 PM UTC

Update timeline

  1. investigating Apr 29, 2024, 08:02 PM UTC

    We're experiencing ingest delay on sessions and errors of <15 minutes in the worst case due to index bloat on some postgres tables.

  2. identified Apr 29, 2024, 08:02 PM UTC

    Ingest delay is <15 minutes and decreasing as we optimize some of the bloated indexes.

  3. monitoring Apr 29, 2024, 11:19 PM UTC

    Ingest delay is now <1 minute and continuing to trend down.

  4. resolved Apr 30, 2024, 12:35 AM UTC

    No backlog; index fixes are in place.