Balena Outage History
Balena is up right nowPingoru has recorded 51 incidents for Balena: 20 in the last 12 months and 3 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 0 | 325 of 720 (45%) |
| May 2026 | 1 | 3 of 744 (0%) |
| June 2026 | 0 | 0 of 720 (0%) |
| July 2026 | 3 | 174 of 744 (23%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 1 | 2 of 720 (0%) |
| October 2026 | 1 | 1 of 522 (0%) |
Incidents, newest first
Minor October 19, 2026 Elevated API Errors 1h API, Dashboard
Started Oct 19, 2026, 06:30 PM UTC· resolved Oct 19, 2026, 07:30 PM UTC
- investigating Oct 19, 2026, 06:30 PM UTC
We're experiencing an elevated level of API errors (device logs) and are currently looking into the issue.
- monitoring Oct 19, 2026, 06:50 PM UTC
We have identified the cause and have taken measures for system restoring, we are monitoring and seeing improvements on logs stability and reporting
- resolved Oct 19, 2026, 07:30 PM UTC
This incident has been resolved.
- postmortem Oct 20, 2026, 08:32 PM UTC
A change to the caching mechanism inadvertently omitted our logs backend read-context handling, causing customer log queries to fail. Log ingestion was unaffected, and no logs were lost.
Major September 28, 2026 Elevated Dashboard Errors 1h 50m API, Dashboard
Started Sep 28, 2026, 04:23 PM UTC· resolved Sep 28, 2026, 06:14 PM UTC
- investigating Sep 28, 2026, 04:23 PM UTC
We're experiencing an elevated level of errors on our Dashboard and are currently looking into the issue.
- identified Sep 28, 2026, 04:46 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 28, 2026, 04:50 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 28, 2026, 06:14 PM UTC
This incident has been resolved.
- postmortem Sep 29, 2026, 02:18 PM UTC
On September 28, the balenaCloud dashboard failed to load because some API requests stopped responding. A runtime upgrade to our API interacted badly with a legacy storage library, causing requests that depended on it to hang. We rolled back the upgrade, which restored service. To prevent this from happening again, we are removing the legacy code path involved and adding customizable timeouts for external backends.
Minor July 27, 2026 Elevated Error on Public Device Url 1h 52m Device URLs
Started Jul 27, 2026, 04:49 PM UTC· resolved Jul 27, 2026, 06:41 PM UTC
- investigating Jul 27, 2026, 04:49 PM UTC
We're currently seeing some Gateway error on public device url. Investigation ongoing.
- investigating Jul 27, 2026, 05:06 PM UTC
We are continuing to investigate this issue.
- monitoring Jul 27, 2026, 05:06 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 27, 2026, 06:41 PM UTC
Unusual load on the proxy, caused some degraded performances on the device URL. Everything is now working properly.
Minor March 31, 2026 Elevated GIT/Application Builder Errors 21d 3h Application Builder
Started Mar 31, 2026, 12:55 PM UTC· resolved Apr 21, 2026, 04:30 PM UTC
- identified Mar 31, 2026, 12:55 PM UTC
We're experiencing an elevated level of errors in our application builder infrastructure and are currently looking into the issue.
- monitoring Apr 08, 2026, 07:51 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Apr 08, 2026, 07:51 PM UTC
We are continuing to monitor for any further issues.
- resolved Apr 21, 2026, 04:30 PM UTC
This incident has been resolved.
- postmortem Apr 21, 2026, 05:06 PM UTC
Starting around March 11, some cloud builds began failing intermittently with no such image errors. The failures were non-deterministic and affected all architectures. At peak, some users saw around 50% failure rates. We identified and fixed several bugs in the builder's image garbage collector that caused it to over-count freed disk space and run too aggressively, eventually deleting images that in-progress builds still needed. Fixes were deployed between March 19 and April 14, with build failure rates dropping to near-zero after the final deploy. We're continuing to monitor and working on additional safeguards to prevent the garbage collector from targeting images that active builds depend on.
Minor March 23, 2026 Builder Degraded performance 1d 20h Application Builder
Started Mar 23, 2026, 04:57 PM UTC· resolved Mar 25, 2026, 01:18 PM UTC
- investigating Mar 23, 2026, 04:57 PM UTC
We are seeing several builds intermitently failing with 404 errors - No such image during builds and are investigating.
- monitoring Mar 23, 2026, 06:55 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 25, 2026, 01:18 PM UTC
This incident has been resolved.
- postmortem Mar 25, 2026, 03:38 PM UTC
Between March 11 and March 25, some cloud builds experienced intermittent failures with "no such image" errors. The issue was non-deterministic and did not affect all builds. We've identified a likely contributing factor and deployed mitigations that have stabilized build reliability. We're continuing to investigate the underlying cause to prevent recurrence. If you experienced build failures during this window, re-running your build should succeed. We appreciate your patience while we worked through this, and we apologize for the disruption.
Minor February 26, 2026 Elevated Device SSH Errors 8h 18m SSH proxy
Started Feb 26, 2026, 07:20 PM UTC· resolved Feb 27, 2026, 03:39 AM UTC
- investigating Feb 26, 2026, 07:20 PM UTC
We're experiencing an elevated level of device SSH errors and are currently looking into the issue.
- investigating Feb 26, 2026, 09:13 PM UTC
We are continuing to investigate this issue. It only affects the direct SSH to device connection for example established via the balena-cli with balena device ssh . Web Terminal via the dashboard and standard tunnelling is not affected.
- identified Feb 27, 2026, 02:41 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Feb 27, 2026, 03:02 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 27, 2026, 03:39 AM UTC
This incident has been resolved.
- postmortem Mar 17, 2026, 11:47 AM UTC
On February 26, 2026 at approximately 19:00 UTC, users began experiencing failures when connecting to their devices via balena device ssh. The command returned a generic "host error" with no further detail. The issue affected all CLI-based SSH access through the balena proxy. Web terminal access and balena device tunnel remained functional throughout the incident and were available as workarounds. Devices themselves were online and healthy; the failure occurred at the authentication stage between the proxy and the device. **Root cause** The balena proxy runs several internal processes. As a resource-saving measure, the service that handles web terminal connections had been disabled in the proxy pods dedicated to CLI SSH, since those pods do not serve web terminal sessions. What was not recognized at the time was that this service also carried the responsibility for loading SSH identity keys into the shared SSH agent used by the proxy. With it disabled, the proxy pods had no keys loaded and could not authenticate with devices, causing every SSH connection attempt to fail at the authentication stage. **Resolution** The service that handles web terminal connections was temporarily re-enabled in the SSH proxy pods while a permanent fix was being developed, restoring SSH key availability and resolving the authentication failures. A permanent fix has since been deployed. **Follow-up** We have decoupled the SSH key loading from the service that handles web terminal connections so that it is an independent initialization process in the proxy. We also added additional monitoring to ensure that SSH access through the proxy is adequately checked. We sincerely apologize for the impact this caused. We are committed to improving our processes to prevent issues like these.
Major February 10, 2026 Elevated Delta Errors 14h 28m Delta Image Downloads
Started Feb 10, 2026, 09:33 AM UTC· resolved Feb 11, 2026, 12:02 AM UTC
- investigating Feb 10, 2026, 09:33 AM UTC
Some delta generation requests are encountering errors and failing. We are currently investigating this issue.
- monitoring Feb 10, 2026, 10:25 AM UTC
We have identified the potential cause and have rolled back the changes.
- resolved Feb 11, 2026, 12:02 AM UTC
This incident has been resolved.
- postmortem Feb 11, 2026, 12:10 AM UTC
v2 delta generation service experienced failures from ~21:15 UTC Feb 9 to ~10:00 UTC Feb 10, 2026, due to a missing configuration dependency during a logic change. **Impact:** * v2 delta generation requests failed to complete * No data loss or security impact **Root Cause:** Recent logic changes were deployed without the required accompanying configuration update, preventing the service from completing v2 delta requests. **Resolution:** The logic changes were rolled back, restoring the service to its previous stable state. **Follow-up Actions:** * Prepare and deploy the permanent fix We apologize for the disruption and any inconvenience this caused. We are committed to improving our processes to prevent similar issues in the future.
Critical February 10, 2026 Elevated Cloudlink Errors 3h 51m Cloudlink (VPN)
Started Feb 10, 2026, 04:05 AM UTC· resolved Feb 10, 2026, 07:56 AM UTC
- investigating Feb 10, 2026, 04:05 AM UTC
We're experiencing an elevated level of errors in our Cloudlink infrastructure and are currently looking into the issue.
- identified Feb 10, 2026, 06:39 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Feb 10, 2026, 07:17 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 10, 2026, 07:56 AM UTC
This incident has been resolved.
- postmortem Feb 10, 2026, 11:52 AM UTC
Balena devices were unable to connect to Cloudlink on February 10, 2026, from approximately 02:26 GMT to 07:11 GMT due to an expired server certificate. Devices that were already connected to Cloudlink were unaffected unless the connection was terminated. **Root Cause:** The Cloudlink servers were using an expired certificate that was due for replacement. Consequently, incoming Cloudlink connections failed with a certificate verification error. **Resolution:** The certificate has been replaced, and Cloudlink servers were restarted to use the new certificate. Balena devices are expected to reconnect to Cloudlink within a few minutes after being disconnected due to the restart. **Follow-up Actions:** * Expand certificate expiry monitoring coverage to include all active certificates * Automate the certificate renewal process for Cloudlink We apologize for any disruption this caused and appreciate your patience as we continue improving our processes and operations.
Major January 28, 2026 Partial API Outage 25m
Started Jan 28, 2026, 12:35 AM UTC· resolved Jan 28, 2026, 01:00 AM UTC
- resolved Jan 28, 2026, 01:40 AM UTC
A configuration change causing multiple component restarts resulted in the API being unavailable for a few minutes.
- postmortem Feb 26, 2026, 10:58 AM UTC
Between 00:35 GMT and 01:00 GMT on January 28, 2026, users may have experienced intermittent connectivity issues with the balenaCloud API. Consequently, users may have had difficulty using the balenaCloud dashboard, and some devices also faced delays in reporting their current state to the backend. **Root Cause:** The incident was triggered during a planned configuration update to our environment. This update caused a brief, simultaneous restart of several core service components. As these services resumed, a high volume of concurrent device Cloudlink reconnections created a temporary surge in traffic. This surge exceeded the immediate processing capacity of our API backend, leading to increased latency and a temporary reduction in available service instances as they struggled to clear the request queue. **Resolution:** Our engineering team monitored the API as it processed the initial backlog. Service stabilized as the surge of reconnection requests subsided and API instances returned to healthy operating levels. Normal service was fully restored by 01:00 GMT. **Follow-up Actions:** We are reviewing the API deployment configuration and performance thresholds to better handle rapid spikes in traffic. We sincerely apologize for the disruption to your workflow and appreciate your continued patience as we improve the resilience of our platform.
Minor December 23, 2025 Degraded Performance 17h 6m API, Application Builder, balenahub
Started Dec 23, 2025, 04:32 PM UTC· resolved Dec 24, 2025, 09:38 AM UTC
- investigating Dec 23, 2025, 04:32 PM UTC
We are currently investigating an issue affecting the availability of balenaCloud services.
- monitoring Dec 23, 2025, 07:03 PM UTC
Scaling issues during service deployment caused by unavailability of nodes from underlying scaling provider.
- resolved Dec 24, 2025, 09:38 AM UTC
Insufficient AWS compute capacity overloaded the remaining nodes. This high load caused readiness probes to fail, triggering API restarts that created a feedback loop of increasing pressure.
Critical December 5, 2025 An upstream provider outage is affecting connectivity to balenaCloud services 31m API, Dashboard, Website
Started Dec 05, 2025, 09:05 AM UTC· resolved Dec 05, 2025, 09:37 AM UTC
- identified Dec 05, 2025, 09:05 AM UTC
CloudFlare, our proxy provider, is having service issues. Connectivity to balenaCloud services are currently affected.
- monitoring Dec 05, 2025, 09:16 AM UTC
Our upstream provider has implemented some fixes. balenaCloud services are back online. We are still monitoring the situation.
- resolved Dec 05, 2025, 09:37 AM UTC
This incident has been resolved.
Minor November 12, 2025 Builder partially degraded service 10h 45m Application Builder
Started Nov 12, 2025, 09:13 AM UTC· resolved Nov 12, 2025, 07:58 PM UTC
- investigating Nov 12, 2025, 09:13 AM UTC
We are currently investigating this issue.
- identified Nov 12, 2025, 03:01 PM UTC
We have observed packet loss in the connections to our builder workers. We are working with our upstream provider to fix the issue. We have also provisioned a few replacement workers in an alternative location while we mitigate the problem.
- monitoring Nov 12, 2025, 05:14 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 12, 2025, 07:58 PM UTC
This incident has been resolved.
- postmortem Nov 12, 2025, 08:08 PM UTC
### Summary On November 12, 2025, we experienced an incident causing increased build failures. This was traced to a significant network issue upstream of our hosting provider, which was outside their direct control. Build success rates returned to normal as of 17:15 UTC, and we formally closed the incident at 20:00 UTC after a period of extended monitoring. ### What Happened A network issue between upstream service providers resulted in severe packet loss. This network degradation disrupted connections to our remote builder workers, causing an increase in builds failing with connection timeout errors \(e.g., `ETIMEDOUT`\). ### Our Response & Mitigation * While the upstream network issue was being addressed, we brought additional builder workers online in unaffected regions to successfully route and process builds. * We were in communication with our hosting provider, who was monitoring the external network problem. ### Resolution * **Recovery \(17:15 UTC\):** The upstream network provider resolved the issue, and we observed build failure rates and connection quality return to normal levels. * **Incident Closed \(20:00 UTC\):** After an extended period of monitoring to ensure all systems remained stable, we formally closed the incident. We apologize for the disruption this caused.
Notice October 1, 2025 Elevated Device URLs/VPN Errors –
Started Oct 01, 2025, 01:30 PM UTC· resolved Oct 01, 2025, 01:30 PM UTC
- resolved Oct 01, 2025, 02:51 PM UTC
An update to our kube-system infrastructure resulted in a disruptive pod rollout that left some devices disconnected from Cloudlink for up to 10 minutes. We apologize for the interruption and future updates to this component will be handled within planned maintenance windows.
Major September 30, 2025 Elevated Device VPN Tunnel Errors 2h 8m Cloudlink (VPN)
Started Sep 30, 2025, 07:47 PM UTC· resolved Sep 30, 2025, 09:56 PM UTC
- investigating Sep 30, 2025, 07:47 PM UTC
We're experiencing an elevated level of errors in our Device VPN Tunnel infrastructure and are currently looking into the issue.
- identified Sep 30, 2025, 08:35 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 30, 2025, 09:34 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 30, 2025, 09:56 PM UTC
This incident has been resolved.
- postmortem Oct 01, 2025, 12:51 PM UTC
On **September 30th**, following a production deployment of a core component, a critical authorization failure affected the `balena device tunnel` command, preventing users from establishing port tunnel connections. ### Summary of the Incident A subtle bug was introduced several weeks ago in a **small database query change** within a component. This change passed our standard review process and all pre-deployment testing, including unit tests utilizing mocked API endpoints. Because of other pending changes, this component update was not immediately deployed. When it was finally released to production on September 30th, the query exhibited an unexpected incompatibility with the **live production API environment**. The failure was not immediately apparent through our primary monitoring, but once the authorization issue was identified, our team quickly found the flawed query, deployed a patched component, and restored full functionality. ### Root Cause and Timeline * **Change Introduction:** A small change to a database query was merged several weeks ago. * **Failed Validation:** The change passed unit tests and code review but failed to correctly interact with the real-world production API due to a subtle environmental or data-specific condition. * **Deployment & Failure:** On **September 30th**, the component was deployed. The authorization failure for `balena device tunnel` was subsequently observed. * **Resolution:** The bug was quickly diagnosed, the query was patched, and a fixed component was deployed to production, resolving the incident. ### Corrective Actions We are taking immediate steps to prevent this type of failure from recurring: 1. **Unit Test Realism:** We have **updated our unit test mocks** to more accurately reflect invalid or non-standard production responses, ensuring future query changes are validated against real-world failure modes. 2. **End-to-End Test Scheduling:** We are prioritizing the development and scheduling of a new **end-to-end test** specifically dedicated to validating the full functionality of the `device tunnel` command in a production-like environment. This will catch integration errors sooner. We apologize for the interruption this caused to your workflow. We are committed to using this incident to improve the robustness and reliability of our continuous deployment process.
Notice July 7, 2025 Degraded API performance –
Started Jul 07, 2025, 10:00 AM UTC· resolved Jul 07, 2025, 10:00 AM UTC
- resolved Jul 07, 2025, 10:40 AM UTC
Temporarily degraded performance of API response processing.
Minor April 17, 2025 Degraded Performance on device state endpoints 21h 47m API
Started Apr 17, 2025, 02:38 PM UTC· resolved Apr 18, 2025, 12:26 PM UTC
- investigating Apr 17, 2025, 02:38 PM UTC
We are currently investigating this issue.
- identified Apr 17, 2025, 02:56 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 17, 2025, 03:00 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 18, 2025, 12:26 PM UTC
This incident has been resolved.
Minor April 10, 2025 Unable to pin releases in the dashboard 1h 40m Dashboard
Started Apr 10, 2025, 01:00 PM UTC· resolved Apr 10, 2025, 02:40 PM UTC
- investigating Apr 10, 2025, 01:00 PM UTC
We are currently investigating an issue with users unable to pin releases in the dashboard.
- identified Apr 10, 2025, 01:00 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 10, 2025, 01:59 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 10, 2025, 02:40 PM UTC
This incident has been resolved.
- postmortem Apr 14, 2025, 12:52 PM UTC
A UI update included updates to a few nested dependencies, which changed a behavior we relied on. The result was preventing the data population of available releases in the Target Release section on the Fleet Summary page. We initially reverted the changed package, and subsequently changed our approach so that it works with the latest version of the package.
Minor March 22, 2025 Degraded cloudlink connection performance 30m Device URLs, Cloudlink (VPN)
Started Mar 22, 2025, 12:10 AM UTC· resolved Mar 22, 2025, 12:40 AM UTC
- identified Mar 22, 2025, 12:10 AM UTC
The issue has been identified and a fix is being implemented.
- identified Mar 22, 2025, 12:10 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Mar 22, 2025, 12:37 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Mar 22, 2025, 12:37 AM UTC
We are continuing to monitor for any further issues.
- resolved Mar 22, 2025, 12:40 AM UTC
This incident has been resolved.
Minor February 5, 2025 Elevated API Log Errors 21h 13m API, Dashboard
Started Feb 05, 2025, 04:45 PM UTC· resolved Feb 06, 2025, 01:58 PM UTC
- investigating Feb 05, 2025, 05:51 PM UTC
We're experiencing an elevated level of log stream errors and are currently looking into the issue.
- monitoring Feb 05, 2025, 05:53 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Feb 05, 2025, 05:53 PM UTC
We are continuing to monitor for any further issues.
- resolved Feb 06, 2025, 01:58 PM UTC
This incident has been resolved.
- postmortem Feb 06, 2025, 07:26 PM UTC
We deployed an upgrade to our log system that worked well in staging, but in production hit issues with log streams so we quickly rolled it back. We have identified the issue and why it only affected one environment, and are investigating an alternate solution going forward.
Critical December 24, 2024 Elevated Device URL Errors 34m Device URLs
Started Dec 24, 2024, 11:06 AM UTC· resolved Dec 24, 2024, 11:40 AM UTC
- investigating Dec 24, 2024, 11:06 AM UTC
We're experiencing an elevated level of Device URL errors and are currently looking into the issue.
- monitoring Dec 24, 2024, 11:35 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Dec 24, 2024, 11:40 AM UTC
This incident has been resolved.
Notice December 12, 2024 The OS versions list for upgrading HosOS throught the dashboard does not show any version for some devices. –
Started Dec 12, 2024, 09:00 PM UTC· resolved Dec 12, 2024, 09:00 PM UTC
- resolved Dec 13, 2024, 02:31 AM UTC
The OS versions list under the settings menu of some devices may not be showing any versions for some devices preventing users from upgrading their HostOS. Some users may also see a notice on their device summary page saying "OS downgrades are not allowed". We have reverted the dashboard to a previous version that has a working OS version list for upgrading HostOS. We are still investigating why the OS version list is not rendered in the latest dashboard version.
Notice November 12, 2024 Elevated Delta Image Downloads Errors –
Started Nov 12, 2024, 09:06 AM UTC· resolved Nov 12, 2024, 09:06 AM UTC
- resolved Nov 12, 2024, 09:06 AM UTC
The delta image back-end was failing to connect to our internal worker nodes because it was using an outdated authentication certificate. The back-end and the worker nodes were to be configured to use a new certificate at the same time to avoid disruptions but we had a delay in re-configuring the back-end. We are reviewing the process to avoid more disruptions like this in the future.
Notice October 27, 2024 Balena API reporting outdated device heartbeat status –
Started Oct 27, 2024, 01:00 AM UTC· resolved Oct 27, 2024, 01:00 AM UTC
- resolved Oct 28, 2024, 01:20 PM UTC
Some devices may have the status of "Reduced Functionality" on the dashboard because the API returned outdated Heartbeat status. We found that the error occurred after we switched to a new cache for the API backend. We applied a fix to the API to allow it to display the latest reported heartbeat status from the devices. We are still investigating how to prevent this error from occurring in the future.