Balena experienced a minor incident on May 18, 2025 affecting Cloudlink (VPN), lasting 3h 4m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 18, 2025, 01:13 PM UTC
We're experiencing an elevated level of errors in our Device URLs and VPN infrastructure and are currently looking into the issue.
- identified May 18, 2025, 01:26 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 18, 2025, 01:30 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 18, 2025, 04:17 PM UTC
This incident has been resolved.
- postmortem May 20, 2025, 06:25 PM UTC
We were notified of an issue with memory consumption on some of the Cloudlink pods in our cluster on Sunday. We were able to redeploy the pods while we investigated the root cause. There should have been little or no interruption of service while the new pods came online and the old ones were slowly drained. We have since implemented one improvement to avoid memory consumption issues, and are looking into other possible root causes.