Trustgrid Public experienced a major incident on December 7, 2021 affecting Cloud Controller, lasting 10h 42m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 07, 2021, 05:36 PM UTC
AWS is experiencing outages of several services that are potentially impacting Trustgrid services. We are actively monitoring to determine if changes are needed to Trustgrid services to respond.
- monitoring Dec 07, 2021, 06:11 PM UTC
We are attempting to set the control plane to a read-only state out of an abundance of caution. However the AWS management services are currently degraded which is preventing this change. We will continue to monitor and update as appropriate.
- monitoring Dec 07, 2021, 06:30 PM UTC
We have successfully set the control plan to "read-only." We will restore as soon as AWS stabilizes.
- monitoring Dec 07, 2021, 08:40 PM UTC
We are continuing to monitor but AWS is not reporting improvements. We remain in read-only state for the time being. Changes can be made in the portal but will not be published to the nodes.
- monitoring Dec 07, 2021, 10:30 PM UTC
AWS seems to be recovering but is still reporting some issues. We will hold in this state until we see them fully recovered and stable.
- monitoring Dec 08, 2021, 04:18 AM UTC
Underlying AWS services are behaving normally. We've re-enabled configuration changes to nodes and are fully operational.
- resolved Dec 08, 2021, 04:18 AM UTC
This incident has been resolved.