Blacksmith incident

Blacksmith control plane outage

Major Resolved

Blacksmith experienced a major incident on May 11, 2026 affecting Blacksmith Managed Runners (eu-central ARM) and Blacksmith Managed Runners (eu-central x86) and 1 more component, lasting 6h 35m. The incident has been resolved; the full update timeline is below.

Started
May 11, 2026, 03:56 PM UTC
Resolved
May 11, 2026, 10:31 PM UTC
Duration
6h 35m
Detected by Pingoru
May 11, 2026, 03:56 PM UTC

Affected components

Blacksmith Managed Runners (eu-central ARM)Blacksmith Managed Runners (eu-central x86)Blacksmith Managed Runners (us-west ARM)Blacksmith Managed Runners (us-west x86)Blacksmith Managed Runners (eu-west x86)Blacksmith Managed Runners (us-central MacOS)

Update timeline

  1. investigating May 11, 2026, 03:56 PM UTC

    There may be some delays with job adoption

  2. monitoring May 11, 2026, 03:58 PM UTC

    We implemented a fix and are re-queuing jobs in the backlog

  3. monitoring May 11, 2026, 07:32 PM UTC

    We are still investigating issues with slow job adoption.

  4. identified May 11, 2026, 08:30 PM UTC

    We are continuing to face issues with our upstream database provider. We are working with them to root cause the problem.

  5. resolved May 11, 2026, 10:31 PM UTC

    This incident has been resolved. Queue times are back to normal, we are working on mitigations on our end to prevent such load based degradations in the future.