Highlight.io incident

Delayed session and error ingest

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on December 23, 2024, lasting 16h 20m. The incident has been resolved; the full update timeline is below.

Started
Dec 23, 2024, 11:18 PM UTC
Resolved
Dec 24, 2024, 03:38 PM UTC
Duration
16h 20m
Detected by Pingoru
Dec 23, 2024, 11:18 PM UTC

Update timeline

  1. investigating Dec 23, 2024, 11:18 PM UTC

    We're experiencing a data ingest delay for sessions and errors due to Kafka maintenance. We're working on a fix. Traces and logs are currently ingested with no delay.

  2. investigating Dec 24, 2024, 03:06 PM UTC

    We are investigating a disk space issue in the cluster.

  3. resolved Dec 24, 2024, 03:38 PM UTC

    We have resolved a disk space issue that caused our kafka cluster to become unhealthy.