Trustgrid Public Outage History

We determined the issue was related to a reference to a dependency that is no longer needed. AWS issues caused the internal authentication service to fail, but new instances could not start because that dependency no longer exists. We were able to remove the reference and have brought the service back online.

The Cloudflare network outage has the potential to prevent appliance node upgrades and agent node installs/upgrades from completing successfully. We are monitoring Cloudflare's status page and will update once it is resolved. In the meantime we recommend avoiding triggering upgrades or deploying new agents.

We are currently unable to process and display flow logs due to an ongoing issue affecting AWS services. Our systems remain operational otherwise. We are monitoring the situation closely, and flow log processing will automatically resume once AWS resolves the underlying issue.

In addition to the disruption affecting flow log processing, we are observing intermittent errors within the Trustgrid Portal. These typically resolve after refreshing the page. We continue to monitor AWS’s ongoing incident, and normal operation of both flow logs and portal access is expected to resume once their issues are resolved.

Due to ongoing AWS instability, we recommend pausing any non-essential configuration changes in the Trustgrid Portal. This will help avoid potential errors or incomplete updates until AWS services stabilize. We’ll provide another update once conditions improve.

We’re continuing to monitor the ongoing AWS service issues impacting flow log processing and causing intermittent portal errors. The situation is largely unchanged. As a precaution, please avoid any non-essential configuration changes until AWS stabilizes. We’ll share another update as soon as we have new information.

AWS has reported continued signs of recovery, but we are keeping this incident open until their issue is fully resolved. No further updates are planned unless the situation changes or AWS confirms full restoration.

Earlier today, we received reports of multiple users encountering intermittent 502 Bad Gateway errors when attempting to log in. In most cases, the errors were temporary, though it’s possible some users experienced persistent issues. Initial investigation points to an issue with the AWS WAF configuration as the likely cause. We have applied updates to the WAF settings and are actively monitoring the system to ensure stability. We’ll continue to track for any recurrence and will follow up with additional details once our investigation is complete.

We continue to monitor the situation but our load balancers have not seen a 502 error since the configuration change.

The issue appears to have resolved. We have confirmed the same control plane service had an issue as on February 14 and we are confirming the same root cause. For clarification, there is no indication that this impacting data plane traffic. Only connectivity to the Trustgrid control plane.

We are still seeing a large number of nodes showing disconnected after the control plane service stablized. We are taking steps to attempt to resolve this.