Blacksmith incident

Job adoption delays

Major Resolved

Blacksmith experienced a major incident on June 17, 2026 affecting Blacksmith Managed Runners (eu-central ARM) and eu-central x86 and 1 more component, lasting 1h 3m. The incident has been resolved; the full update timeline is below.

Started
Jun 17, 2026, 12:45 PM UTC
Resolved
Jun 17, 2026, 01:48 PM UTC
Duration
1h 3m
Detected by Pingoru
Jun 17, 2026, 12:45 PM UTC

Affected components

Blacksmith Managed Runners (eu-central ARM)eu-central x86us-west ARMus-west x86eu-west x86us-central MacOSGithub → ActionsGithub → Webhooks

Update timeline

  1. investigating Jun 17, 2026, 12:45 PM UTC

    We are receiving reports of job adoption delays, we are currently investigating.

  2. investigating Jun 17, 2026, 01:01 PM UTC

    We believe this is related to an issue with Github Webhooks and are still investigating.

  3. identified Jun 17, 2026, 01:14 PM UTC

    We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.

  4. monitoring Jun 17, 2026, 01:27 PM UTC

    Our engineers are implementing a mitigation which should take 20 minutes to deploy. Thank you for your patience.

  5. monitoring Jun 17, 2026, 01:48 PM UTC

    We implemented a fix and are currently monitoring the result but seeing recovery. Any existing queued jobs wont self recover so please be sure to cancel and rerun those jobs.

  6. resolved Jun 17, 2026, 01:48 PM UTC

    This incident has been resolved.