Highlight.io experienced a minor incident on April 30, 2024, lasting 8h 57m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Apr 30, 2024, 02:27 PM UTC
We're experiencing ingest delay on sessions and errors of ~60 minutes in the worst case due to increased error inference load. We're investigating for a fix.
- identified Apr 30, 2024, 03:24 PM UTC
Ingest backlog is currently at <30 minutes.
- identified Apr 30, 2024, 05:12 PM UTC
We're performing an emergency scale up to our postgres database which may cause a brief outage of highlight loading. Trace / log alerts continue to work as expected.
- monitoring Apr 30, 2024, 05:21 PM UTC
Frontend is stable now that we are on the larger postgres database instance.
- monitoring Apr 30, 2024, 09:35 PM UTC
We're tracking down to a much lower backlog now at <15 minutes.
- resolved Apr 30, 2024, 11:25 PM UTC
Caught up on backlog.