RAM Tracking experienced a major incident on October 24, 2023, lasting 20h 59m. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Oct 26, 2023, 09:06 AM UTC
At 15:55 UTC on Tuesday (24/10/2023), the processing of camera events on RAM Live ran into an unexpected issue. Our messaging processor lost contact with the messaging queue, bringing processing to a halt. The issue was identified at 08:00 UTC Wednesday morning and message processing was brought back online at 12:54 UTC. Unfortunately the messaging queue retained messages for only 1 hour, meaning some journeys have not been processed. As part of our internal post-mortem, we have increased the retention period to 24 hours to avoid this happening again.