Highlight.io experienced a minor incident on May 4, 2024, lasting 19h 11m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating May 04, 2024, 01:33 AM UTC
Our latest maintenance window configured our Kafka Connector consumers to optimize cpu / memory usage, but the autoscaling configuration is resulting in OOM crashes. We're reverting those changes now.
- monitoring May 04, 2024, 02:04 AM UTC
Configuration addressed; backlogs are decreasing now.
- monitoring May 04, 2024, 02:16 AM UTC
Logs fully caught up.
- monitoring May 04, 2024, 02:44 AM UTC
Traces backlog is now <1 hr and decreasing.
- monitoring May 04, 2024, 02:45 AM UTC
ETA for traces catch up is 30 minutes from now.
- resolved May 04, 2024, 08:45 PM UTC
Traces delay resolved at 20:30 PT as of May 3rd.