Firebolt experienced a minor incident on June 29, 2021 affecting Tenant Account Creation, lasting 7h 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 29, 2021, 07:56 PM UTC
Tenant account creation fails due to AWS ACM increased API latency and increased API error rates. Official response from AWS support: We can confirm increased API latency and increased API error rates for the ACM APIs in the US-EAST-1 Region. During this time, you may be unable to Request new certificates, and may also observe errors when attempting to List and/or modify existing certificates. This issue impacts both the AWS Management Console, and the ACM APIs. Additionally, you may also receive API errors when attempting to associate new resources. Existing associated resources are unaffected, and continue to operate as normal. We have identified the root cause of the issue and are working toward mitigation and resolution. We will provide further updates as we have more information to share.
- identified Jun 29, 2021, 07:56 PM UTC
The issue has been identified and a fix is being implemented.
- identified Jun 29, 2021, 07:57 PM UTC
We are continuing to work on a fix for this issue.
- identified Jun 29, 2021, 08:18 PM UTC
Tenant account creation fails due to AWS ACM increased API latency and increased API error rates.
- identified Jun 29, 2021, 08:29 PM UTC
AWS continues to investigate the issue. 1:14 PM PDT We continue to work toward mitigating the affected subsystem responsible for the increase in API Errors and Latencies for the ACM APIs. Other AWS Services (such as ClientVPN) who attempt to create or associate new certificates may also be impacted by this issue. Existing resources remain unaffected by this issue and continue to operate normally. https://status.aws.amazon.com/
- resolved Jun 30, 2021, 05:34 AM UTC
AWS has fixed the issue on their side. AWS response: We have identified the limit issue and have mitigated it. The issue has been fully resolved and all ACM API requests are being answered normally. During this time, all existing resources that had a configured ACM certificate (such as ELB load balancers and CloudFront distributions) continued to operate normally, and were not impaired by this issue.