Highlight.io experienced a minor incident on May 3, 2024, lasting 31m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating May 03, 2024, 07:56 PM UTC
We're noticing sudden load on our ClickHouse cluster causing a ~5 minute delay in ingested data. We're investigating
- identified May 03, 2024, 08:09 PM UTC
Repeated polling from the logs page caused a number of long running queries to take over the cluster. We've killed the queries and are working on a fix to prevent this going further. Backlogged data is being written now.
- identified May 03, 2024, 08:10 PM UTC
Traces delay: 21 minutes Logs delay: 14 minutes
- identified May 03, 2024, 08:14 PM UTC
Traces delay: 15 minutes Logs delay: <1 minute
- monitoring May 03, 2024, 08:25 PM UTC
Traces delay: 3 minutes Logs delay: <1 minute
- resolved May 03, 2024, 08:27 PM UTC
Caught up on all data.