Highlight.io experienced a minor incident on April 22, 2024, lasting 5h 57m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Apr 22, 2024, 05:29 PM UTC
We've just gotten an internal alert for a trace ingest delay of >30 minutes (and increasing). Investigating an increased load on our ClickHouse database.
- identified Apr 22, 2024, 10:10 PM UTC
we have a periodic 'fetch more' query that happens that's causing really high load. That query is causing a lot of load on clickhouse.
- identified Apr 22, 2024, 10:10 PM UTC
We've deployed a workaround now to disable the query with these filters. We've found an issue with our frontend which is increasing the date range for the query.
- monitoring Apr 22, 2024, 10:11 PM UTC
The ingest delay for traces is now ~14 minutes and decreasing as we catch up on the backlog.
- resolved Apr 22, 2024, 11:26 PM UTC
We've now caught up on the backlog and resolved the issue.