Elevated Device SSH Errors

Update timeline

1. investigating Feb 24, 2026, 05:27 PM UTC

   We're experiencing an elevated level of device SSH errors and are currently looking into the issue.

2. investigating Feb 24, 2026, 05:29 PM UTC

   We are continuing to investigate this issue.

3. identified Feb 24, 2026, 06:29 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Feb 24, 2026, 06:41 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Feb 24, 2026, 08:54 PM UTC

   This incident has been resolved.

6. postmortem Feb 25, 2026, 06:42 PM UTC

   On Feb 24, 2026 around ~17:20 UTC, a routine infrastructure deployment caused intermittent availability issues with Device URLs and web terminal access. Devices remained online and functional throughout, and CLI-based SSH access was unaffected. The issue was caused by a configuration change that intentionally disabled several internal services no longer required by our proxy infrastructure. However, these services were still associated with pod health checks. A misconfigured override mechanism applied this change to production before it had passed through all required release gate checks, which would have caught the failing health checks. The issue was identified quickly through automated monitoring and service was restored manually while a permanent fix was deployed. We have since corrected the underlying configuration override mechanism and are adding additional monitoring coverage to catch similar issues before they reach production. We apologize for the disruption and thank you for your patience.