Trustgrid Public experienced a major incident on February 14, 2025 affecting Cloud Controller, lasting 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 14, 2025, 04:41 PM UTC
We are currently investigating what appears to be a control plane issue causing nodes to show as disconnected.
- investigating Feb 14, 2025, 04:53 PM UTC
We have seen a large number of nodes reconnect. So far we've seen no indication this impacted data plane connectivity for customers. We continue to investigate.
- investigating Feb 14, 2025, 05:09 PM UTC
We continue to investigate but the issue appears to have resolved itself.
- resolved Feb 14, 2025, 05:34 PM UTC
Identified Issue: We have identified a bug that caused a container to crash unexpectedly. As a result, nodes temporarily appeared as disconnected until their connection timed out and re-established with a new container. Current Status: The issue has been identified, and we are implementing a fix to prevent future occurrences. Nodes have automatically reconnected, and normal operations have resumed. Next Steps: Our team is investigating the root cause and applying necessary updates to improve container stability. We apologize for any disruptions and appreciate your patience. If you experience continued issues, please reach out to our support team.