Balena incident

Elevated API Log Errors

Minor Resolved View vendor source →

Balena experienced a minor incident on February 5, 2025 affecting API and Dashboard, lasting 21h 13m. The incident has been resolved; the full update timeline is below.

Started
Feb 05, 2025, 04:45 PM UTC
Resolved
Feb 06, 2025, 01:58 PM UTC
Duration
21h 13m
Detected by Pingoru
Feb 05, 2025, 04:45 PM UTC

Affected components

APIDashboard

Update timeline

  1. investigating Feb 05, 2025, 05:51 PM UTC

    We're experiencing an elevated level of log stream errors and are currently looking into the issue.

  2. monitoring Feb 05, 2025, 05:53 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring Feb 05, 2025, 05:53 PM UTC

    We are continuing to monitor for any further issues.

  4. resolved Feb 06, 2025, 01:58 PM UTC

    This incident has been resolved.

  5. postmortem Feb 06, 2025, 07:26 PM UTC

    We deployed an upgrade to our log system that worked well in staging, but in production hit issues with log streams so we quickly rolled it back. We have identified the issue and why it only affected one environment, and are investigating an alternate solution going forward.