AI-Media Outage History

Started Sep 23, 2026, 11:30 PM UTC· resolved Sep 23, 2026, 11:30 PM UTC

1. resolved Sep 23, 2026, 11:30 PM UTC

   On September 23, 2026, some existing jobs using Falcon Classic for legacy HTTP POST workflows stopped delivering captions due to an issue introduced during a planned platform update. Other Falcon workflows and services were not affected. Impact: Some Falcon Classic jobs that were already running were interrupted between approximately 1:54 PM ET and 6:45 PM ET. Affected jobs may have shown as running but were not delivering captions. New jobs started between 2:19 PM ET and 3:11 PM ET were also at risk. Impact to new jobs was minimal. Resolution: All affected jobs were moved back to a stable environment, and caption delivery has been fully restored. We are reviewing our update process to prevent this from happening again. If you are still experiencing issues with a Falcon Classic job, please contact support.

Read the full incident report →

Started Jul 29, 2026, 07:00 PM UTC· resolved Jul 29, 2026, 07:00 PM UTC

1. resolved Jul 30, 2026, 09:03 PM UTC

   At approximately 3:15 PM EDT on July 29, an AWS host-level failure caused one Lexi worker instance to be replaced automatically. Because the instance was terminated forcefully, jobs running on it could not be requeued and were left in a state where they appeared active but were no longer producing captions. 16 jobs were affected, and captioning resumed on restart. All affected jobs have been cleared, and Lexi is operating normally.

Read the full incident report →

Started Jul 24, 2026, 11:00 AM UTC· resolved Jul 24, 2026, 11:00 AM UTC

1. resolved Jul 30, 2026, 08:43 PM UTC

   At 06:56 AM EDT on July 24, an AWS routing issue in us-west-2 caused several client connections to drop across two US relays within a three minute window. All affected clients reconnected cleanly to other US relays. Sydney relays were unaffected. Connection stability returned to normal following the reconnects.

Read the full incident report →

Started Jul 24, 2026, 11:00 AM UTC· resolved Jul 25, 2026, 02:00 AM UTC

1. resolved Jul 30, 2026, 08:44 PM UTC

   On July 24, an expired backend credential left the iCap Connection Stability and iCap Server Cluster Reconnects metrics blank for approximately 15 hours. This overlapped with the relay disconnects reported separately, so customers checking these metrics that morning had no visibility into connection health. Metrics for this window show a gap.

Read the full incident report →

Started Jan 29, 2026, 10:59 PM UTC· resolved Jan 30, 2026, 01:43 AM UTC

1. investigating Jan 29, 2026, 10:59 PM UTC

   We are currently experiencing an issue with LEXI connecting to encoders set with a preference to use the relay 2 server. Effected encoders will likely not be able see LEXI captions during this time. Our team is working to identify the cause and will share updates shortly. If your encoder is impacted, please remove the relay preference in iCap Admin (https://accounts.eegicap.com/) under the encoder user account and recreate the LEXI instance. Please reach out to support for further assistance as needed.

2. identified Jan 30, 2026, 01:00 AM UTC

   A cause has been identified and a fix is being tested to resolve any ongoing issues. Thanks for your patience. Please contact support if you are still impacted by this. Thank you.

3. monitoring Jan 30, 2026, 01:13 AM UTC

   A fix is deployed and we are seeing relay 2 service return to normal. We are continuing monitor. Please contact support if you are encountering any issues. Thank you.

4. resolved Jan 30, 2026, 01:43 AM UTC

   The deployed fix has been successful and we are marking this as resolved. We will continue to monitor to ensure stability. Please contact support if any further assistance is required. Thank you for your patience as this was resolved.

5. postmortem Feb 05, 2026, 02:13 AM UTC

   # **Post Incident Review LEXI/iCap Outage \(29 January 2026\)** ## **Overview:** **Incident Title:** Relay 2 LEXI Connection Issue – 29 January 2026 **Date & Time \(UTC\):** Jan 29, 18:07 - Jan 30, 01:00 **Date & Time \(AEDT\):** Jan 30, 05:07 - Jan 30, 12:00 **Date & Time \(EST\):** Jan 29, 13:07 - Jan 29, 20:00 **Affected Service: iCap, LEXI** **Impact Scope: Global** **Severity: High** **Duration: 6 hours 53 minutes** ‌ ## **Summary of Incident:** On January 29, 2026, a deployment to Relay 2 \(13.52.203.48\) completed at 18:07 UTC. Following this deployment, encoders configured to prefer relay\_2 began experiencing connection failures, with the first customer report received at 20:06 UTC. As the issue progressed, the Ai-Media Support team began receiving additional cases, including reports that LEXI jobs were not running. This indicated that the impact extended beyond encoder connectivity and affected downstream captioning workflows as well. The root cause was identified as a deployment script that unintentionally blocked traffic from certain IPs to relay\_2 while other relays remained unaffected. The blocks were removed at 01:00 UTC \(30 Jan\), restoring service, and customers confirmed normal operation shortly thereafter ## **Timeline of Events \(UTC\):** ## **Impact:** * **Duration:** 6 hours 53 minutes * **Services Affected:** iCap/LEXI **Clients Impacted:** * Encoder and LEXI connections became stuck in disconnected state. * The issue impacted a small subset of customers—between 1% and 5%—and only those whose encoders were configured to Relay 2. ‌ ## **Root Cause Analysis \(RCA\):** A script used during deployment unintentionally applied blocking rules that affected a subset of customer IP ranges — **but only on relay\_2**, not on the other relay servers. This caused encoder/LEXI connections set to prefer relay\_2 to fail to connect. ## **Resolution:** At **01:00 UTC — 30 January 2026**, the technical team **removed the traffic blocks** that had been unintentionally applied by the deployment script on **relay\_2**, and initial testing immediately showed that the issue was resolved. ## **Corrective Actions & Improvements:** **Immediate Fixes** * Removal of the traffic blocks from Relay 2. * Validation of successful encoder reconnections. **Short Term Improvements** * Add deployment script auditing before execution. * Implement diff-check or rule-compare for firewall changes per relay. * Update internal runbooks to include: * Early detection of relay specific behavior * Explicit post-deploy traffic validation **Long Term Prevention** * Introduce automated configuration validation pipelines for relay nodes. * Enforce “canary-style” deployment \(1 relay first, monitor results\). * Improve monitoring to detect unusual connection failure patterns quickly. * Build alerting on encoder disconnect spikes for any specific relay. ‌ ## **Commitment to Clients:** We recognize the impact this incident had on our clients’ operations and deeply regret the disruption. AI-Media is committed to: * Maintaining transparent communication during and after incidents. * Strengthening our technical safeguards to reduce the likelihood of similar outages. * Continuously improving our disaster recovery capabilities to ensure service reliability.

Read the full incident report →

Started Jan 08, 2026, 12:12 AM UTC· resolved Jan 08, 2026, 04:07 AM UTC

1. investigating Jan 08, 2026, 12:12 AM UTC

   We are currently investigating a potential issue impacting the connection and performance of the US East Falcon server. If you are experiencing any Falcon issues on US East, please create a new Falcon Instance and change the location from US East to US West while we investigate. Please reach out to support if you need assistance. Thank you.

2. monitoring Jan 08, 2026, 01:16 AM UTC

   A fix has been applied and we are seeing normal service resume on Falcon US East connections. We are continuing to monitor to confirm this is resolved.

3. resolved Jan 08, 2026, 04:07 AM UTC

   US East has maintained normal service since the fix was deployed and we are marking this as resolved. We are continuing to monitor and ensuring robust solutions are in place to continue consistent performance. Please reach out to support if you encounter any issues. Thank you.

Read the full incident report →

Started Dec 09, 2025, 05:34 AM UTC· resolved Dec 10, 2025, 09:25 AM UTC

1. resolved Dec 09, 2025, 05:34 AM UTC

   The root cause of the incident impacting iCap and LEXI on the 8th of December has been identified. The iCap primary server became unresponsive for a short period which prompted an auto failover of load to the secondary server. The secondary server became overloaded by the surge of connections. A manual restart of the primary server resolved the issue. A Post Incident Report is underway and is on track to be published to this Status Page by 5pm AEDT.

2. postmortem Dec 09, 2025, 10:55 PM UTC

   ## **Post Incident Review Lexi/iCap Outage \(8 December 2025\)** ‌ **Overview:** **Incident Title:** LEXI/iCap Outage – 8 December 2025 **Date & Time \(AEDT\):** 18:49 - 20:25 **Date & Time \(EST\):** 2:49 - 4:25 **Date & Time \(UTC\):** 7:49 - 9:25 **Affected Service:** iCap, Lexi **Impact Scope:** Global **Duration:** 96 minutes **Severity:** Critical ‌ **Summary of Incident:** On Monday 8 December 2025 at 07:49 UTC \(02:49 EST / 6:49pm AEDT\), iCap/LEXI services experienced a global outage, resulting in an inability to provide automated captioning and related services. Our teams implemented human captioning where possible to minimize client impact. Services were restored at 9:25 UTC, with full verification completed before confirming resolution. ‌ **Timeline of Events \(UTC\):** * **7:49** – Outage detected by service delivery coordinators. * **7:49 – 8:20** – Initial troubleshooting and escalation; human captioning initiated in APAC where possible to mitigate disruption. * **9:05 –** Technical investigation identified root cause. * **9:10 – 9:15** - Fix implemented and verified by technical teams. * **9:25** – Service restoration confirmed; impacted clients notified. ‌ **Impact:** * **Duration:** 96 minutes * **Services Affected:** LEXI/iCap Live automated captioning, translation, and voice services * **Clients Impacted:** Multiple broadcast and enterprise clients globally. ‌ **Root Cause:** The iCap primary server became unresponsive for a short period which prompted an auto failover of load to the secondary server, in accordance with standard failover procedures. The secondary server became overloaded by the surge of connections. The primary server auto recovered _partially_ however it could not effectively communicate its health status to the secondary backup server to allow it to resume load. ‌ **Resolution:** A manual restart resulted in full health restoration of the primary server which triggered the load being effectively transferred from the secondary backup server to the primary. At full health the primary server resumed services load effectively. ‌ **Corrective Actions & Improvements:** 1. Capacity review and uplift of both the primary and secondary server 2. Upgrades to the communication process between the primary production server and the secondary backup server 3. Optimization of auto alerting ‌ **Commitment to Clients:** We recognize the impact this incident had on our clients’ operations and deeply regret the disruption. AI-Media is committed to: * Maintaining transparent communication during and after incidents. * Strengthening our technical safeguards to reduce the likelihood of similar outages. * Continuously improving our disaster recovery capabilities to ensure service reliability.

Read the full incident report →

Started Dec 08, 2025, 07:49 AM UTC· resolved Dec 08, 2025, 09:30 AM UTC

1. investigating Dec 08, 2025, 09:24 AM UTC

   We are currently experiencing an issue with LEXI and iCap. Captioning sessions are impacted and users will may be unable to access LEXI and iCap during this time. Our team is working to identify the cause and will share updates shortly.

2. resolved Dec 08, 2025, 09:45 AM UTC

   LEXI and iCap incident is now resolved. Incident duration was approximately 96 minutes. A fix was applied at 9:10 UTC and service restoration confirmed at approximately 9:25 UTC. Support teams are monitoring. Root cause analysis and remediation actions will be investigated and a Post Incident Report published by 5pm Thursday AEDT.

3. postmortem Dec 11, 2025, 05:11 AM UTC

   **Post Incident Review Lexi/iCap Outage \(8 December 2025\)** **Overview:** **Incident Title:** LEXI/iCap Outage – 8 December 2025 **Date & Time \(AEDT\):** 18:49 - 20:25 **Date & Time \(EST\):** 2:49 - 4:25 **Date & Time \(UTC\):** 7:49 - 9:25 **Affected Service:** iCap, Lexi **Impact Scope:** Global **Duration:** 96 minutes **Severity:** Critical ‌ **Summary of Incident:** On Monday 8 December 2025 at 07:49 UTC \(02:49 EST / 6:49pm AEDT\), iCap/LEXI services experienced a global outage, resulting in an inability to provide automated captioning and related services. Our teams implemented human captioning where possible to minimise client impact. Services were restored at 9:25 UTC, with full verification completed before confirming resolution. **Timeline of Events \(UTC\):** * **7:49** – Outage detected by service delivery coordinators. * **7:49 – 8:20** – Initial troubleshooting and escalation; human captioning initiated in APAC where possible to mitigate disruption. * **9:05 –** Technical investigation identified root cause. * **9:10 – 9:15** - Fix implemented and verified by technical teams. * **9:25** – Service restoration confirmed; impacted clients notified. **Impact:** * **Duration:** 96 minutes * **Services Affected:** LEXI/iCap Live automated captioning, translation, and voice services * **Clients Impacted:** Multiple broadcast and enterprise clients globally. **Root Cause** The iCap primary server became unresponsive for a short period which prompted an auto failover of load to the secondary server, in accordance with standard failover procedures. The secondary server became overloaded by the surge of connections. The primary server auto recovered _partially_ however it could not effectively communicate its health status to the secondary backup server to allow it to resume load. **Resolution:** A manual restart resulted in full health restoration of the primary server which triggered the load being effectively transferred from the secondary backup server to the primary. At full health the primary server resumed services load effectively. **Corrective Actions & Improvements:** 1. Capacity review and uplift of both the primary and secondary server 2. Upgrades to the communication process between the primary production server and the secondary backup server 3. Optimization of auto alerting **Commitment to Clients:** We recognize the impact this incident had on our clients’ operations and deeply regret the disruption. AI-Media is committed to: * Maintaining transparent communication during and after incidents. * Strengthening our technical safeguards to reduce the likelihood of similar outages. * Continuously improving our disaster recovery capabilities to ensure service reliability.

Read the full incident report →

Started Nov 10, 2025, 11:25 PM UTC· resolved Nov 11, 2025, 02:12 AM UTC

1. investigating Nov 10, 2025, 11:25 PM UTC

   We are currently experiencing an issue with Ai-Live Lexi session in the AU region. Users are unable to commence Lexi during this time. Our team is working to identify the cause and will share updates shortly.

2. identified Nov 10, 2025, 11:50 PM UTC

   We have identified the potential issue and are now implementing a fix. Thank you for your patience as we test. We will update again shortly.

3. monitoring Nov 11, 2025, 01:29 AM UTC

   We have implemented a fix and seeing some improvement. We will continue to monitor

4. resolved Nov 11, 2025, 02:12 AM UTC

   We have seen successful connections return to Ai-Live LEXI from 10:45am AEST / 6:45pm EST in the AU region as services resume. If you encounter any persisting issues, please contact support.

Read the full incident report →

Started Nov 07, 2025, 12:00 PM UTC· resolved Nov 08, 2025, 12:00 AM UTC

1. resolved Nov 14, 2025, 08:12 PM UTC

   Date: November 7, 2025, approximately 7:00 PM - 7:03 PM ET Status: Resolved Summary: We experienced a brief service disruption to LEXI Live due to a sudden traffic spike. Some users experienced delayed connections or temporary disconnections. Impact: New transcription jobs experienced delays of up to 2-5 minutes in establishing connections Some active transcription sessions experienced brief disconnections (ranging from 48 seconds to 2+ minutes) Root Cause: A rapid spike in traffic exceeded our standard capacity, causing a temporary backlog while our autoscaling systems provisioned additional capacity. Resolution: Service automatically recovered as additional capacity came online. All systems returned to normal operations shortly after. Next Steps: We are implementing several improvements to our auto-scaling infrastructure. These include faster response times to traffic spikes and increased baseline capacity to better handle peak usage periods.

Read the full incident report →

Started Nov 06, 2025, 04:00 PM UTC· resolved Nov 06, 2025, 09:00 PM UTC

1. resolved Nov 07, 2025, 05:15 PM UTC

   The Ai-Live US instance experienced an outage at just before 11:00am EST at which time clients reported the Ai-Live Viewer link was down. The issue was caused by unexpected Viewer load in multiple Ai-Live sessions. Ai-Live server resources were added at 11:35am EST in order to accommodate the increased load. Additional resources were enabled at 3:51 and 3:57pm EST to mitigate increasing CPU utilization as viewership continued to grow. The issue was fully resolved at 4:00pm EST.

Read the full incident report →

Started Oct 20, 2025, 06:58 PM UTC· resolved Oct 20, 2025, 11:07 PM UTC

1. investigating Oct 20, 2025, 06:58 PM UTC

   We are currently experiencing an issue affecting EEGCloud services. We are investigating the problem.

2. monitoring Oct 20, 2025, 08:08 PM UTC

   As AWS's services are coming back online, Lexi services are also recovering. We will continue to monitor and take action as needed. If you continue to experience any problems, please contact support

3. resolved Oct 20, 2025, 11:07 PM UTC

   Lexi services have stabilized and we are comfortable marking this incident as resolved. All systems are operating normally following the AWS outage. If you experience any issues, please contact support.

Read the full incident report →

Started Oct 10, 2025, 02:30 AM UTC· resolved Oct 10, 2025, 03:15 AM UTC

1. resolved Oct 10, 2025, 03:59 AM UTC

   One of our iCap server nodes carrying iCap and Lexi traffic became unresponsive and required manual intervention to restore. Other servers should have been able to fulfill most jobs but some temporary interruptions occurred due to transfers, retries, and related conditions. Issues began about 02:35 UTC and were clear at 03:15 UTC.

Read the full incident report →

Started Sep 29, 2025, 09:56 PM UTC· resolved Sep 29, 2025, 10:00 PM UTC

1. resolved Oct 06, 2025, 10:30 PM UTC

   We investigated a brief service disruption that affected LEXI and Translate jobs on September 29th (21:56-22:00 UTC). The disruption lasted approximately 4 minutes and 19 seconds and was related to an automatic update of an AWS service performed during that time. Users experienced different impacts depending on when their jobs were initiated: New LEXI/Translate jobs started during this window were unable to connect to the AWS service during startup, resulting in startup failures. These jobs self-resolved once the disruption ended. LEXI/Translate jobs already running before the disruption experienced significant caption delays. Captions produced during the disruption were output in a large batch after service restoration, with read-speed functionality automatically increasing output speed to compensate before returning to normal real-time operation.

2. postmortem Oct 07, 2025, 11:23 PM UTC

   **LEXI Disruption Postmortem:** September 29, 2025, 21:56:04 - 22:00:23 UTC \(4m 19s\) **Summary:** A 4-minute and 19-second service disruption affected LEXI and Translate jobs due to an automatic update of a critical backend service. New jobs failed to start, while running jobs experienced significant caption delays that self-resolved after the disruption ended. **Impact** * **New Jobs:** Unable to connect to backend service during startup, resulting in startup failures. Jobs self-resolved when the disruption ended. * **Running Jobs:** Experienced significant caption delays during the disruption window. Delayed captions were output in batch after service restoration, with read-speed functionality automatically compensating before returning to normal operation. **Root Cause:** An automatic update of a critical backend service during the maintenance window caused the temporary unavailability of a dependency required for LEXI/Translate job initialization and real-time caption processing. **Timeline** * 21:56:04 UTC - Disruption begins with backend service update * 21:56:04 - 22:00:23 UTC - New jobs fail to start; running jobs experience delays * 22:00:23 UTC - Backend service update completes; service functionality restored * Post-incident - Investigation and remediation planning initiated **Resolution** Service was automatically restored once the backend service update was completed. No manual intervention was required. **Action Items Completed** * Modified service update window to minimize user impact * Improved fallback handling mechanisms for backend service dependencies **Action Items In Progress** * Implementing additional redundancy measures to strengthen service resilience during dependency updates

Read the full incident report →

Started Aug 09, 2025, 06:25 AM UTC· resolved Aug 09, 2025, 01:09 PM UTC

1. investigating Aug 09, 2025, 11:25 AM UTC

   Starting at approximately 2:25AM ET on 8/9/25, the Lexi system began experiencing difficulties allowing new jobs to be created. This is a critical issue that is actively being investigated by our team. We will update as soon as we have more information.

2. monitoring Aug 09, 2025, 12:23 PM UTC

   Our team has implemented a fix and we are monitoring.

3. resolved Aug 09, 2025, 01:09 PM UTC

   Our team believes this to be resolved as of 8:10AM ET. We are still closely monitoring and an investigation into the root cause is still ongoing.

4. postmortem Aug 19, 2025, 03:11 PM UTC

   At 2:22am EDT / 6:22am UTC on August 9, an AI-Media staff member performing a deployment to a staging environment inadvertently reconfigured a routing table used by the production LEXI Live system. This resulted in all active, cloud-based LEXI Live jobs- as well as any jobs started thereafter- being unable to generate captions until the condition was rectified at 8:10am EDT / 12:10pm UTC. Additionally, the outage did not correctly trigger failover to LEXI DR systems, except where users had jobs scheduled in EEG Cloud; scheduled jobs failed over correctly. Further work is planned to prevent similar disruptions going forward.

Read the full incident report →

Started Aug 04, 2025, 11:36 PM UTC· resolved Aug 05, 2025, 12:24 AM UTC

1. investigating Aug 04, 2025, 11:36 PM UTC

   It has been noticed that the Agent Portal is currently inaccessible.

2. investigating Aug 04, 2025, 11:39 PM UTC

   We are continuing to investigate this issue.

3. identified Aug 05, 2025, 12:01 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Aug 05, 2025, 12:18 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Aug 05, 2025, 12:24 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 24, 2025, 05:07 PM UTC· resolved Jul 24, 2025, 05:07 PM UTC

1. resolved Jul 24, 2025, 05:07 PM UTC

   We want to sincerely apologize for any confusion caused by recent emails you may have received, indicating the cancellation or expiration of your subscription. These messages were triggered in error by a recent administrative adjustment to our billing system. The emails may have included details such as: - A cancellation date of 24-Jul-2025 - Number of unpaid invoices - Amount due Please rest assured: these notifications can be safely disregarded. They do not reflect the actual status of any subscriptions, and there is no impact on product access or usage. All active subscriptions remain fully functional, and this incident is resolved. We apologize for the undue concern this may have caused, and thank you for your understanding.

Read the full incident report →

Started Jun 26, 2025, 01:52 PM UTC· resolved Jun 26, 2025, 03:32 PM UTC

1. investigating Jun 26, 2025, 01:52 PM UTC

   We have received some reports this morning that users are unable to create new Falcon instances. We are currently investigating. The Falcon service remains operational and existing Falcon instances should be unaffected.

2. resolved Jun 26, 2025, 03:32 PM UTC

   We have restored full functionality to Falcon and confirmed users are able to create new Falcon instances again. Marking this incident as resolved. If you have any questions about this incident please reach out to our Support team at: [email protected]

Read the full incident report →

Started Apr 24, 2025, 12:30 AM UTC· resolved Apr 24, 2025, 01:45 AM UTC

1. resolved Apr 24, 2025, 02:09 PM UTC

   Beginning around 8:26 pm EST (23 Apr) / 00:26 UTC (24 Apr), Lexi's scheduler experienced an outage lasting approximately 70 minutes. Events scheduled to start in that window did not start automatically. Events scheduled to end only terminated if the instance had a timeout set, and only after the job reached that timeout. After investigating, we were able to trace this issue to a recent update to Lexi's scheduling database. This has been resolved and Lexi's scheduler is fully functional. Please direct any questions about this incident to our Support team at [email protected].

Read the full incident report →

Started Apr 23, 2025, 06:02 PM UTC· resolved Apr 24, 2025, 01:51 PM UTC

1. monitoring Apr 23, 2025, 06:02 PM UTC

   Earlier today, around 11:37 am EST / 15:37 UTC, iCap experienced a brief outage. Service was restored to 100% after a few minutes. We are currently monitoring to make sure this incident is not ongoing.

2. monitoring Apr 23, 2025, 08:52 PM UTC

   Update: we are continuing to investigate for a root cause on today's iCap outage. Monitoring shows all iCap metrics are reporting as normal and we do not believe this to be an ongoing issue.

3. resolved Apr 24, 2025, 01:51 PM UTC

   After monitoring for the past day, we can confirm service has been fully restored and iCap is operating normally. More info to come in a post-mortem.

Read the full incident report →

Started Apr 03, 2025, 04:00 AM UTC· resolved Apr 04, 2025, 04:30 AM UTC

1. resolved Apr 03, 2025, 02:42 PM UTC

   Beginning around 23:54 April 2 EST / 03:54 April 3 UTC, and lasting approximately 30 minutes, several iCap servers in the US-WEST-2 region experienced a brief issue with internet connectivity. Users connected to the affected servers were automatically moved to other servers in the cluster. We have traced this outage to an infrastructure misconfiguration related to the system maintenance earlier in the day on April 2. This issue was resolved around 12:30 AM ET / 4:30 UTC April 3 and the iCap system remains fully operational since that time.

Read the full incident report →

Started Dec 10, 2024, 10:00 AM UTC· resolved Dec 10, 2024, 12:30 PM UTC

1. resolved Dec 11, 2024, 04:57 PM UTC

   Beginning around 5 AM EST / 10:00 UTC and lasting until around 7:30AM EST / 12:30 UTC on Dec 10, one of the iCap relay servers experienced a period of instability. This may have resulted in disruptions or delays in audio and caption delivery to connected users on this server. Our engineering team worked to restore the server to its normal operating state and manually rebalance some of the load to minimize customer impact. Further work is planned to prevent similar disruptions going forward.

Read the full incident report →

Started Nov 26, 2024, 04:30 PM UTC· resolved Nov 26, 2024, 06:30 PM UTC

1. resolved Nov 27, 2024, 12:18 AM UTC

   On Nov 26/2024 - from 16:30 UTC to 18:30 UTC - Scheduled Lexi and/or Translate jobs failed to start during this period. This issue has been identified and resolved, and no further occurrences have been observed. Lexi/Translate jobs scheduled during this window will remain unaffected and start succesfully.

Read the full incident report →

Started Nov 21, 2024, 06:11 PM UTC· resolved Nov 21, 2024, 07:59 PM UTC

1. investigating Nov 21, 2024, 06:11 PM UTC

   We have received reports that the Lexi scheduler is inaccessible and/or failing to start scheduled events. We are investigating and will report back here once we have more information.

2. monitoring Nov 21, 2024, 06:19 PM UTC

   Our engineering team has implemented a hotfix to restore access to the Lexi scheduler. We are monitoring to ensure everything is operating normally.

3. resolved Nov 21, 2024, 07:59 PM UTC

   Lexi's scheduling functionality has been fully restored and continues to operate as expected. Marking this incident as resolved, post-mortem to follow.

4. postmortem Nov 21, 2024, 11:26 PM UTC

   ## What happened? Today around 12:05pm ET / 17:05 UTC, we deployed an update to our production Lexi environment, specifically to address the scheduling system. Unfortunately this deployment contained a configuration error which made it unable to communicate with the database that stores Lexi scheduled events. This prevented scheduled Lexi events from starting, and also prevented users from accessing the Lexi Scheduling page on [eegcloud.tv](http://eegcloud.tv). Once we discovered this issue, we rolled back the change, which restored functionality. Lexi scheduling was fully operational again by 1:16pm ET / 18:16 UTC. ## What are we doing about it? Going forward we will: * Review & update our testing procedures in order to identify problems like this before they're deployed * Implement additional alerts so we can be notified more quickly about any production-impacting issues ‌Ai-Media understands the importance of a reliable cloud service for closed captioning and we apologize for the disruption this incident caused to Lexi users. If there are any follow-up questions on this incident, please submit a ticket to [[email protected]](mailto:[email protected]).

Read the full incident report →

Started Oct 30, 2024, 02:45 PM UTC· resolved Oct 30, 2024, 03:52 PM UTC

1. investigating Oct 30, 2024, 02:45 PM UTC

   We have received reports of some Translate sessions not generating caption data or not starting up normally. We are currently investigating and will report back here with more information.

2. identified Oct 30, 2024, 03:08 PM UTC

   Our engineering team has identified the cause of this issue and are currently working on a fix to get Translate back up & running. We will report back here with more information.

3. resolved Oct 30, 2024, 03:52 PM UTC

   Our engineering team has implemented a fix and Translate sessions should be running normally again. If there are any questions about this incident, please reach out to our support team at [email protected].

Read the full incident report →