nusutus+ experienced a critical incident on December 7, 2021 affecting Live Agent, lasting 8h 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 07, 2021, 04:35 PM UTC
The service provider is currently engaged and have identified the root cause of the issues in the US-EAST-1 Region, which is a network issue with some network devices in that Region which is affecting multiple services, including the console but also other services in US-EAST-1. We are actively working towards recovery.
- investigating Dec 07, 2021, 05:20 PM UTC
This issue is impacting the APIs of multiple AWS services, including EC2, Connect, DynamoDB, QuickSight, as well as the AWS console. We are attempting multiple paths to mitigating this issue, and have all teams engaged. Unfortunately at this time we do not have an ETA to when this issue will be mitigated
- investigating Dec 07, 2021, 06:34 PM UTC
Phone calls are in the process of being re-routed, so calls will bypass the Ping Agent service. Until the AWS service disruption has passed, stores will have to service their own calls.
- identified Dec 07, 2021, 07:33 PM UTC
AWS continues to work on multiple mitigation paths in parallel to address the issue with some network devices in the US-EAST-1. A root cause has been found and the AWS team is focused on resolving as soon as possible. No ETA has been given at this time.
- identified Dec 07, 2021, 10:08 PM UTC
AWS is confirming recovery for service availability in the US-EAST-1 Region, however we are not seeing this recovering in our system at this time. We will continue to monitor and will update the thread as we get live information.
- identified Dec 07, 2021, 11:59 PM UTC
Many of the AWS services have recovered, however AWS is working towards full recovery across services.
- monitoring Dec 08, 2021, 01:08 AM UTC
Testing and monitoring of the AWS connect system continues at this time. We will update you as soon as we have more information.
- resolved Dec 08, 2021, 01:21 AM UTC
The AWS issue impacting our Ping Agent service is now resolved. We will be migrating the service back online over the next couple hours.