Balena experienced a major incident on March 3, 2025 affecting API, lasting 5h 37m. The incident has been resolved; the full update timeline is below.
Affected components
API
Update timeline
- investigating Mar 03, 2025, 06:59 PM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- identified Mar 03, 2025, 07:48 PM UTC
The issue has been identified and a fix is being implemented.
- identified Mar 03, 2025, 08:46 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Mar 03, 2025, 09:03 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 04, 2025, 12:36 AM UTC
This incident has been resolved.
- postmortem Mar 04, 2025, 12:37 AM UTC
An internal observability feature led to unreasonable base memory footprint for API instances under production load, leading to frequent evictions. For now, we’ve rolled back to a previous API version to restore stability, while we investigate the root cause.