Blacksmith incident

GitHub webhooks degraded causing job queueing

Minor Resolved

Blacksmith experienced a minor incident on May 19, 2026 affecting Blacksmith Managed Runners (eu-central ARM) and Blacksmith Managed Runners (eu-central x86) and 1 more component, lasting 8h 49m. The incident has been resolved; the full update timeline is below.

Started
May 19, 2026, 02:50 PM UTC
Resolved
May 19, 2026, 11:39 PM UTC
Duration
8h 49m
Detected by Pingoru
May 19, 2026, 02:50 PM UTC

Affected components

Blacksmith Managed Runners (eu-central ARM)Blacksmith Managed Runners (eu-central x86)Blacksmith Managed Runners (us-west ARM)Blacksmith Managed Runners (us-west x86)Blacksmith Managed Runners (eu-west x86)Blacksmith Managed Runners (us-central MacOS)

Update timeline

  1. investigating May 19, 2026, 02:50 PM UTC

    We're seeing evidence of webhooks delivery being degraded from GitHub. We're investigating.

  2. monitoring May 19, 2026, 03:53 PM UTC

    We're seeing upstream recovery for GitHub webhook deliveries. Jobs may queue as we process GitHub's backlog of webhook events.

  3. monitoring May 19, 2026, 05:10 PM UTC

    We're still seeing a large backlog of queued jobs due to the incident that the system is working through, we're exploring mitigations.

  4. monitoring May 19, 2026, 08:02 PM UTC

    We're still working through a substantial backlog of queued tasks that has accumulated over this period of delayed webhook arrivals.

  5. monitoring May 19, 2026, 09:22 PM UTC

    We are close to the end of the backlog of the queued tasks and are seeing full recovery in certain runner pools.

  6. resolved May 19, 2026, 11:39 PM UTC

    This incident has been resolved.