Blacksmith incident

Job adoption delays

Major Resolved

Blacksmith experienced a major incident on June 17, 2026 affecting Blacksmith Managed Runners (eu-central ARM) and Blacksmith Managed Runners (eu-central x86) and 1 more component, lasting 1h 3m. The incident has been resolved; the full update timeline is below.

Started
Jun 17, 2026, 12:45 PM UTC
Resolved
Jun 17, 2026, 01:48 PM UTC
Duration
1h 3m
Detected by Pingoru
Jun 17, 2026, 12:45 PM UTC

Affected components

Blacksmith Managed Runners (eu-central ARM)Blacksmith Managed Runners (eu-central x86)Blacksmith Managed Runners (us-west ARM)Blacksmith Managed Runners (us-west x86)Blacksmith Managed Runners (eu-west x86)Blacksmith Managed Runners (us-central MacOS)Github → ActionsGithub → Webhooks

Update timeline

  1. investigating Jun 17, 2026, 12:45 PM UTC

    We are receiving reports of job adoption delays, we are currently investigating.

  2. investigating Jun 17, 2026, 01:01 PM UTC

    We believe this is related to an issue with Github Webhooks and are still investigating.

  3. identified Jun 17, 2026, 01:14 PM UTC

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  4. monitoring Jun 17, 2026, 01:27 PM UTC

    Our engineers are implementing a mitigation which should take 20 minutes to deploy. Thank you for your patience.

  5. monitoring Jun 17, 2026, 01:48 PM UTC

    We implemented a fix and are currently monitoring the result but seeing recovery. Any existing queued jobs wont self recover so please be sure to cancel and rerun those jobs.

  6. resolved Jun 17, 2026, 01:48 PM UTC

    This incident has been resolved.