Blacksmith incident

Webhook Service Outage

Major Resolved

Blacksmith experienced a major incident on May 12, 2026 affecting Blacksmith Managed Runners (eu-central ARM) and Blacksmith Managed Runners (eu-central x86) and 1 more component, lasting 3h 7m. The incident has been resolved; the full update timeline is below.

Started
May 12, 2026, 08:39 PM UTC
Resolved
May 12, 2026, 11:46 PM UTC
Duration
3h 7m
Detected by Pingoru
May 12, 2026, 08:39 PM UTC

Affected components

Blacksmith Managed Runners (eu-central ARM)Blacksmith Managed Runners (eu-central x86)Blacksmith Managed Runners (us-west ARM)Blacksmith Managed Runners (us-west x86)Blacksmith Managed Runners (eu-west x86)Blacksmith Managed Runners (us-central MacOS)

Update timeline

  1. investigating May 12, 2026, 08:39 PM UTC

    We are currently investigating this incident.

  2. monitoring May 12, 2026, 08:42 PM UTC

    We implemented a fix and are currently monitoring the result.

  3. resolved May 12, 2026, 11:46 PM UTC

    We have been running into load issues in our database layer that's causing sporadic instability in job adoption times. We have been actively working with our database provider to improve our systems to handle the recently increased load. We are planning proactive maintenance later this week to address this further.