AI-Media experienced a major incident on November 21, 2024 affecting Lexi, lasting 1h 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 21, 2024, 06:11 PM UTC
We have received reports that the Lexi scheduler is inaccessible and/or failing to start scheduled events. We are investigating and will report back here once we have more information.
- monitoring Nov 21, 2024, 06:19 PM UTC
Our engineering team has implemented a hotfix to restore access to the Lexi scheduler. We are monitoring to ensure everything is operating normally.
- resolved Nov 21, 2024, 07:59 PM UTC
Lexi's scheduling functionality has been fully restored and continues to operate as expected. Marking this incident as resolved, post-mortem to follow.
- postmortem Nov 21, 2024, 11:26 PM UTC
## What happened? Today around 12:05pm ET / 17:05 UTC, we deployed an update to our production Lexi environment, specifically to address the scheduling system. Unfortunately this deployment contained a configuration error which made it unable to communicate with the database that stores Lexi scheduled events. This prevented scheduled Lexi events from starting, and also prevented users from accessing the Lexi Scheduling page on [eegcloud.tv](http://eegcloud.tv). Once we discovered this issue, we rolled back the change, which restored functionality. Lexi scheduling was fully operational again by 1:16pm ET / 18:16 UTC. ## What are we doing about it? Going forward we will: * Review & update our testing procedures in order to identify problems like this before they're deployed * Implement additional alerts so we can be notified more quickly about any production-impacting issues Ai-Media understands the importance of a reliable cloud service for closed captioning and we apologize for the disruption this incident caused to Lexi users. If there are any follow-up questions on this incident, please submit a ticket to [[email protected]](mailto:[email protected]).