AI-Media experienced a critical incident on August 9, 2025 affecting Lexi, lasting 6h 44m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 09, 2025, 11:25 AM UTC
Starting at approximately 2:25AM ET on 8/9/25, the Lexi system began experiencing difficulties allowing new jobs to be created. This is a critical issue that is actively being investigated by our team. We will update as soon as we have more information.
- monitoring Aug 09, 2025, 12:23 PM UTC
Our team has implemented a fix and we are monitoring.
- resolved Aug 09, 2025, 01:09 PM UTC
Our team believes this to be resolved as of 8:10AM ET. We are still closely monitoring and an investigation into the root cause is still ongoing.
- postmortem Aug 19, 2025, 03:11 PM UTC
At 2:22am EDT / 6:22am UTC on August 9, an AI-Media staff member performing a deployment to a staging environment inadvertently reconfigured a routing table used by the production LEXI Live system. This resulted in all active, cloud-based LEXI Live jobs- as well as any jobs started thereafter- being unable to generate captions until the condition was rectified at 8:10am EDT / 12:10pm UTC. Additionally, the outage did not correctly trigger failover to LEXI DR systems, except where users had jobs scheduled in EEG Cloud; scheduled jobs failed over correctly. Further work is planned to prevent similar disruptions going forward.