AI-Media experienced a notice incident on July 9, 2023 affecting iCap and Lexi, lasting 2h 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jul 09, 2023, 01:32 PM UTC
We are aware of an incident that occurred today (July 9) starting around 10:30 UTC and lasting approximately 15 minutes. During this time many iCap users were logged out and some Lexi instances were not producing output. The issue looks to be resolved but we are still investigating root cause.
- resolved Jul 09, 2023, 04:00 PM UTC
After monitoring for several hours we are confident this issue is resolved. Post mortem to follow.
- postmortem Jul 09, 2023, 04:01 PM UTC
Starting at approximately 10:28 UTC and lasting until approx 10:42 UTC on Sunday, July 9, we were experiencing processing backups/overloads on the main iCap authentication system. This affected connectivity for most users who were connected at that time and would have prevented or delayed new iCap logins during this period. This also affected Lexi jobs that were running or scheduled to run during this time, as Lexi depends on the iCap network for functionality. We believe we have identified the trigger for this overload and are working to ensure the stability of the iCap system going forward. We apologize for any inconvenience this outage may have caused. If there are any questions or comments please email [[email protected]](mailto:[email protected]) and reference this incident \(**Services Outage - July 9 10:30 UTC**\) in your subject line, and we will get back to you as soon as possible.