Balena experienced a critical incident on September 17, 2025 affecting Cloudlink (VPN), lasting 1h 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 17, 2025, 08:10 PM UTC
We're experiencing an elevated level of errors in our Device URLs and VPN infrastructure and are currently looking into the issue.
- identified Sep 17, 2025, 08:53 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 17, 2025, 08:54 PM UTC
A fix has been implemented and we are monitoring the results.
- identified Sep 17, 2025, 09:32 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 17, 2025, 09:53 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 17, 2025, 10:01 PM UTC
This incident has been resolved.
- postmortem Sep 18, 2025, 02:18 PM UTC
#### Impact Users were unable to access their devices via SSH through the web-terminal or the CLI. The CLI would return an error: `user does not have permission to access device` An error in our Renovate configuration allowed our automation system to merge and deploy an unintended backend component to production. #### Resolution Our team quickly identified the issue and: 1. Immediately rolled back the component to the previous verified version 2. Restored remote SSH access for all affected devices 3. Corrected the Renovate bot configuration to prevent similar automatic deployments #### Response * **Enhanced deployment controls:** We've restored our automation configuration to ensure all components must pass manual review before production deployment * **Improved monitoring:** We're considering implementing additional alerts to catch similar issues faster * **Process review:** We're reviewing our automated deployment processes to identify other potential gaps We apologize for the disruption and appreciate your patience as we resolved this issue. If you continue to experience any problems, please contact our support team.