Highlight.io experienced a minor incident on March 27, 2024, lasting 21h 39m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Mar 27, 2024, 05:10 PM UTC
As of 03:00 PT we're behind on ingesting session data. We're looking at an OOM with our workers that seems to be causing the issue.
- identified Mar 27, 2024, 07:09 PM UTC
Working on a fix for the delay - ETA for catch up is ~5pm PT.
- identified Mar 27, 2024, 08:28 PM UTC
Fix is out; we're catching up on the delay. ETA to being fully caught up is around 5 hours from now.
- monitoring Mar 27, 2024, 08:37 PM UTC
Scaling our ingest, it looks like we will catch up within 2 hours.
- monitoring Mar 28, 2024, 01:31 AM UTC
Still 1-2 hours behind and slow to catch up. Monitoring to get an ETA to catch up.
- resolved Mar 28, 2024, 05:37 AM UTC
We have caught up on data in most kafka partitions with a few stragglers still catching up. Recent sessions are showing up now.
- resolved Mar 28, 2024, 02:50 PM UTC
Caught up on all data.