Resolved after 2 hours 28 minutes, affecting iCap and Lexi, on July 9, 2023.
Starting at approximately 10:28 UTC and lasting until approx 10:42 UTC on Sunday, July 9, we were experiencing processing backups/overloads on the main iCap authentication system. This affected connectivity for most users who were connected at that time and would have prevented or delayed new iCap logins during this period. This also affected Lexi jobs that were running or scheduled to run during this time, as Lexi depends on the iCap network for functionality. We believe we have identified the trigger for this overload and are working to ensure the stability of the iCap system going forward. We apologize for any inconvenience this outage may have caused. If there are any questions or comments please email [[email protected]](mailto:[email protected]) and reference this incident \(**Services Outage - July 9 10:30 UTC**\) in your subject line, and we will get back to you as soon as possible.
Affected components
Update timeline
- monitoring Jul 09, 2023, 01:32 PM UTC
We are aware of an incident that occurred today (July 9) starting around 10:30 UTC and lasting approximately 15 minutes. During this time many iCap users were logged out and some Lexi instances were not producing output. The issue looks to be resolved but we are still investigating root cause.
- resolved Jul 09, 2023, 04:00 PM UTC
After monitoring for several hours we are confident this issue is resolved. Post mortem to follow.
- postmortem Jul 09, 2023, 04:01 PM UTC
Starting at approximately 10:28 UTC and lasting until approx 10:42 UTC on Sunday, July 9, we were experiencing processing backups/overloads on the main iCap authentication system. This affected connectivity for most users who were connected at that time and would have prevented or delayed new iCap logins during this period. This also affected Lexi jobs that were running or scheduled to run during this time, as Lexi depends on the iCap network for functionality. We believe we have identified the trigger for this overload and are working to ensure the stability of the iCap system going forward. We apologize for any inconvenience this outage may have caused. If there are any questions or comments please email [[email protected]](mailto:[email protected]) and reference this incident \(**Services Outage - July 9 10:30 UTC**\) in your subject line, and we will get back to you as soon as possible.
Your product runs on more than AI-Media
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services