Trustgrid Public experienced a minor incident on August 7, 2021 affecting Cloud Controller and Management Portal, lasting 10h 45m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 07, 2021, 12:55 AM UTC
We are currently investigating an issue with the control plane reporting multiple nodes disconnecting and then reconnecting.
- investigating Aug 07, 2021, 01:05 AM UTC
We are continuing to investigate. At this point it appears these disconnects are false alarms due to a database issue.
- identified Aug 07, 2021, 01:26 AM UTC
It appears AWS is experiencing issues that are impacting part of our control plane leading to the false alarms about nodes disconnecting. Out of an abundance of caution we are taking steps to prevent any changes from being pushed down to nodes.
- identified Aug 07, 2021, 01:57 AM UTC
AWS has identified an issue and is working to resolve. We are monitoring and working to clear disconnect alerts to minimize notification once the issue is resolved.
- identified Aug 07, 2021, 02:49 AM UTC
We are continuing to work on a fix for this issue.
- identified Aug 07, 2021, 04:22 AM UTC
AWS has reported improvements but we are still seeing issues and monitoring for further improvement.
- identified Aug 07, 2021, 06:30 AM UTC
The underlying issue is now resolved - we are bringing services back online and monitoring alerts and notifications.
- identified Aug 07, 2021, 11:34 AM UTC
Everything is operating optimally, and node connection status alerts are working normally again.
- resolved Aug 07, 2021, 11:41 AM UTC
This incident has been resolved.