Balena Outage History

Starting around March 11, some cloud builds began failing intermittently with no such image errors. The failures were non-deterministic and affected all architectures. At peak, some users saw around 50% failure rates. We identified and fixed several bugs in the builder's image garbage collector that caused it to over-count freed disk space and run too aggressively, eventually deleting images that in-progress builds still needed. Fixes were deployed between March 19 and April 14, with build failure rates dropping to near-zero after the final deploy. We're continuing to monitor and working on additional safeguards to prevent the garbage collector from targeting images that active builds depend on.

Between March 11 and March 25, some cloud builds experienced intermittent failures with "no such image" errors. The issue was non-deterministic and did not affect all builds. We've identified a likely contributing factor and deployed mitigations that have stabilized build reliability. We're continuing to investigate the underlying cause to prevent recurrence. If you experienced build failures during this window, re-running your build should succeed. We appreciate your patience while we worked through this, and we apologize for the disruption.

We identified an issue in Dashboard v32.2.0, released on March 2, 2026, where opening the dashboard via a direct link to certain pages \(such as billing or other account management pages\) could result in being unexpectedly redirected to the fleets overview. This was caused by a race condition in our access control logic that made a routing decision before all authorization data had finished loading. The issue was resolved on March 3, 2026 with a fix that ensures the dashboard waits for all access information to be available before determining whether a user can view a page. We understand this was frustrating, particularly for users trying to manage billing or account settings via bookmarked or shared links. We apologize for the disruption and are adding test coverage for direct-link navigation to prevent similar regressions in the future.

On February 26, 2026 at approximately 19:00 UTC, users began experiencing failures when connecting to their devices via balena device ssh. The command returned a generic "host error" with no further detail. The issue affected all CLI-based SSH access through the balena proxy. Web terminal access and balena device tunnel remained functional throughout the incident and were available as workarounds. Devices themselves were online and healthy; the failure occurred at the authentication stage between the proxy and the device. **Root cause** The balena proxy runs several internal processes. As a resource-saving measure, the service that handles web terminal connections had been disabled in the proxy pods dedicated to CLI SSH, since those pods do not serve web terminal sessions. What was not recognized at the time was that this service also carried the responsibility for loading SSH identity keys into the shared SSH agent used by the proxy. With it disabled, the proxy pods had no keys loaded and could not authenticate with devices, causing every SSH connection attempt to fail at the authentication stage. **Resolution** The service that handles web terminal connections was temporarily re-enabled in the SSH proxy pods while a permanent fix was being developed, restoring SSH key availability and resolving the authentication failures. A permanent fix has since been deployed. **Follow-up** We have decoupled the SSH key loading from the service that handles web terminal connections so that it is an independent initialization process in the proxy. We also added additional monitoring to ensure that SSH access through the proxy is adequately checked. We sincerely apologize for the impact this caused. We are committed to improving our processes to prevent issues like these.

v2 delta generation service experienced failures from ~21:15 UTC Feb 9 to ~10:00 UTC Feb 10, 2026, due to a missing configuration dependency during a logic change. **Impact:** * v2 delta generation requests failed to complete * No data loss or security impact **Root Cause:** Recent logic changes were deployed without the required accompanying configuration update, preventing the service from completing v2 delta requests. **Resolution:** The logic changes were rolled back, restoring the service to its previous stable state. **Follow-up Actions:** * Prepare and deploy the permanent fix We apologize for the disruption and any inconvenience this caused. We are committed to improving our processes to prevent similar issues in the future.

Balena devices were unable to connect to Cloudlink on February 10, 2026, from approximately 02:26 GMT to 07:11 GMT due to an expired server certificate. Devices that were already connected to Cloudlink were unaffected unless the connection was terminated. **Root Cause:** The Cloudlink servers were using an expired certificate that was due for replacement. Consequently, incoming Cloudlink connections failed with a certificate verification error. **Resolution:** The certificate has been replaced, and Cloudlink servers were restarted to use the new certificate. Balena devices are expected to reconnect to Cloudlink within a few minutes after being disconnected due to the restart. **Follow-up Actions:** * Expand certificate expiry monitoring coverage to include all active certificates * Automate the certificate renewal process for Cloudlink We apologize for any disruption this caused and appreciate your patience as we continue improving our processes and operations.