Balena experienced a minor incident on October 30, 2024 affecting Application Builder, lasting 1h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 30, 2024, 10:42 AM UTC
We're experiencing an elevated level of errors in our application builder infrastructure and are currently looking into the issue. Some users may experience DNS resolution errors during builds using the balenaCloud builders.
- monitoring Oct 30, 2024, 11:18 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Oct 30, 2024, 11:18 AM UTC
We are continuing to monitor for any further issues.
- resolved Oct 30, 2024, 12:01 PM UTC
This incident has been resolved.
- postmortem Oct 30, 2024, 05:09 PM UTC
An error was introduced in our builder firewall rules that dropped all outbound traffic from the build context. During this time users would see builds start but fail when a RUN instruction included a command that required internet access. We rolled the builders back to a previous release while we investigated the root cause of the bug. A fix is now being reviewed and tested to catch this edge case that takes time to manifest and was missed in original testing.