Fluxx experienced a major incident on December 7, 2021 affecting Grantmaker and Grantmaker, lasting 8h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 07, 2021, 04:43 PM UTC
Fluxx in the US is suffering from ongoing widespread AWS performance and connectivity issues. We are working to understand how to provide better access and increase performance of the system.
- identified Dec 07, 2021, 04:51 PM UTC
AWS Status: (https://status.aws.amazon.com/) 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery.
- identified Dec 07, 2021, 04:54 PM UTC
AWS is now also reporting errors in the APIs used in our primary region From https://status.aws.amazon.com/ We are experiencing elevated error rates for EC2 APIs in the US-EAST-1 region. We have identified root cause and we are actively working towards recovery.
- identified Dec 07, 2021, 05:06 PM UTC
Current AWS Services affected Amazon Connect (N. Virginia) Amazon DynamoDB (N. Virginia) Amazon Elastic Compute Cloud (N. Virginia) AWS Management Console AWS Support Center
- identified Dec 07, 2021, 06:10 PM UTC
From https://status.aws.amazon.com/ We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified the root cause and are actively working towards recovery.
- monitoring Dec 07, 2021, 06:35 PM UTC
From From https://status.aws.amazon.com/ We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified root cause of the issue causing service API and console issues in the US-EAST-1 Region, and are starting to see some signs of recovery. We do not have an ETA for full recovery at this time.
- monitoring Dec 07, 2021, 07:35 PM UTC
from AWS [11:26 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1. The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region. We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time. Root logins for consoles in all AWS regions are affected by this issue, however customers can login to consoles other than US-EAST-1 by using an IAM role for authentication.
- monitoring Dec 07, 2021, 07:53 PM UTC
We are continuing to monitor for any further issues.
- monitoring Dec 07, 2021, 08:47 PM UTC
From AWS: We continue to experience increased API error rates for multiple AWS Services in the US-EAST-1 Region. The root cause of this issue is an impairment of several network devices. We continue to work toward mitigation, and are actively working on a number of different mitigation and resolution actions. While we have observed some early signs of recovery, we do not have an ETA for full recovery. For customers experiencing issues signing-in to the AWS Management Console in US-EAST-1, we recommend retrying using a separate Management Console endpoint (such as https://us-west-2.console.aws.amazon.com/). Additionally, if you are attempting to login using root login credentials you may be unable to do so, even via console endpoints not in US-EAST-1. If you are impacted by this, we recommend using IAM Users or Roles for authentication. We will continue to provide updates here as we have more information to share.
- monitoring Dec 07, 2021, 10:04 PM UTC
The ongoing AWS outage on the US East Coast is affecting Fluxx back-end systems. At this time we are continuing to monitor the AWS situation and will take steps to improve things as soon as possible. Thank you for your understanding.
- monitoring Dec 07, 2021, 11:29 PM UTC
AWS is seeing services return to stability. Fluxx relies on some of the services that are still experiencing impact, but we are seeing service approaching normal. We will continue to monitor the situation and advise when we are fully returned to operation.
- resolved Dec 08, 2021, 01:06 AM UTC
All Services are now operational.