Balena experienced a major incident on May 5, 2026 affecting API and Application Builder and 1 more component, lasting 2h 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 05, 2026, 06:12 PM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- identified May 05, 2026, 06:19 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 05, 2026, 06:43 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 05, 2026, 09:03 PM UTC
This incident has been resolved.
- postmortem May 05, 2026, 09:18 PM UTC
A vulnerability mitigation update required replacing compute cluster nodes, which, when applied, rolled back due to a timeout. This caused some workloads \(API, delta, builder\) to become temporarily unavailable and triggering some undesired secondary effects, including ungracefully terminating a few long lived instances servicing VPN connections. While the rest of the services came back reasonably quickly within a minute or so, it too around one and a quarter hours to re-establish VPN tunnels. A scheduled maintenance will be posted later to perform this update during a planned outage window.