Elevated Device SSH Errors

Update timeline

1. investigating Feb 26, 2026, 07:20 PM UTC

   We're experiencing an elevated level of device SSH errors and are currently looking into the issue.

2. investigating Feb 26, 2026, 09:13 PM UTC

   We are continuing to investigate this issue. It only affects the direct SSH to device connection for example established via the balena-cli with balena device ssh . Web Terminal via the dashboard and standard tunnelling is not affected.

3. identified Feb 27, 2026, 02:41 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Feb 27, 2026, 03:02 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Feb 27, 2026, 03:39 AM UTC

   This incident has been resolved.

6. postmortem Mar 17, 2026, 11:47 AM UTC

   On February 26, 2026 at approximately 19:00 UTC, users began experiencing failures when connecting to their devices via balena device ssh. The command returned a generic "host error" with no further detail. The issue affected all CLI-based SSH access through the balena proxy. Web terminal access and balena device tunnel remained functional throughout the incident and were available as workarounds. Devices themselves were online and healthy; the failure occurred at the authentication stage between the proxy and the device. **Root cause** The balena proxy runs several internal processes. As a resource-saving measure, the service that handles web terminal connections had been disabled in the proxy pods dedicated to CLI SSH, since those pods do not serve web terminal sessions. What was not recognized at the time was that this service also carried the responsibility for loading SSH identity keys into the shared SSH agent used by the proxy. With it disabled, the proxy pods had no keys loaded and could not authenticate with devices, causing every SSH connection attempt to fail at the authentication stage. **Resolution** The service that handles web terminal connections was temporarily re-enabled in the SSH proxy pods while a permanent fix was being developed, restoring SSH key availability and resolving the authentication failures. A permanent fix has since been deployed. **Follow-up** We have decoupled the SSH key loading from the service that handles web terminal connections so that it is an independent initialization process in the proxy. We also added additional monitoring to ensure that SSH access through the proxy is adequately checked. We sincerely apologize for the impact this caused. We are committed to improving our processes to prevent issues like these.