Highlight.io experienced a minor incident on October 1, 2024, lasting 6d 2h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Oct 01, 2024, 05:19 PM UTC
We're currently experiencing a data ingest delay of approximately 1 hour across sessions, errors, logs and traces.
- identified Oct 01, 2024, 06:16 PM UTC
We've identified an issue in our ClickHouse cluster that is affecting ingest performance.
- identified Oct 01, 2024, 06:17 PM UTC
A code change caused our ClickHouse writes to slow down due to new schema introduced. This has now been resolved and we're catching up on delayed data.
- identified Oct 01, 2024, 07:33 PM UTC
Sessions, errors, and logs are no longer delayed. Traces are still delayed by roughly ~30 minutes.
- resolved Oct 07, 2024, 08:07 PM UTC
Resolved as of October 1st, 7pm PT.