DoControl experienced a minor incident on December 7, 2021 affecting AWS lambda-us-east-1 and AWS apigateway-us-east-1, lasting 12h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Dec 07, 2021, 06:16 PM UTC
AWS experiencing issues around US-EAST-1. we are monitoring the issue closely with them. next update within an hour " API Error Rates in US-EAST-1 We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified the root cause and are actively working towards recovery. We have identified root cause of the issue causing service API and console issues in the US-EAST-1 Region, and are starting to see some signs recovery. We do not have an ETA for full recovery at this time. "
- monitoring Dec 08, 2021, 04:42 AM UTC
Update from aws: [3:03 PM PST] Many services have already recovered, however we are working towards full recovery across services. Services like SSO, Connect, API Gateway, ECS/Fargate, and EventBridge are still experiencing impact. Engineers are actively working on resolving impact to these services. [4:35 PM PST] With the network device issues resolved, we are now working towards recovery of any impaired services. We will provide additional updates for impaired services within the appropriate entry in the Service Health Dashboard.
- resolved Dec 08, 2021, 06:50 AM UTC
it seems that most of the issues we saw the last day has been resolved by aws and the error rate is currently low, we are keeping monitoring this closely and reopening a new incident in case we'll find anything