Highlight.io incident

Delayed traces ingest

Minor Resolved View vendor source →

Highlight.io experienced a minor incident on April 29, 2024, lasting 46m. The incident has been resolved; the full update timeline is below.

Started
Apr 29, 2024, 06:35 PM UTC
Resolved
Apr 29, 2024, 07:21 PM UTC
Duration
46m
Detected by Pingoru
Apr 29, 2024, 06:35 PM UTC

Update timeline

  1. investigating Apr 29, 2024, 06:35 PM UTC

    We're seeing a delay in ingesting traces of ~5 minutes and growing due to new load on our ClickHouse cluster. Scaling the cluster while we investigate the root cause of the load.

  2. resolved Apr 29, 2024, 07:21 PM UTC

    We're now at normal <1min ingest latency. Delays never exceeded 5 minutes.