Blacksmith incident

GitHub outage affecting job failures and dashboard errors

Major Resolved

Blacksmith experienced a major incident on August 17, 2026 affecting Github → API Requests and Github → Webhooks and 1 more component, lasting 6h 57m. The incident has been resolved; the full update timeline is below.

Started
Aug 17, 2026, 01:43 PM UTC
Resolved
Aug 17, 2026, 08:40 PM UTC
Duration
6h 57m
Detected by Pingoru
Aug 17, 2026, 01:43 PM UTC

Affected components

Github → API RequestsGithub → WebhooksDashboard

Update timeline

  1. identified Aug 17, 2026, 01:43 PM UTC

    The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests.

  2. identified Aug 17, 2026, 01:46 PM UTC

    Upstream incident has been declared: We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed. We are monitoring and are looking at potential mitigations.

  3. monitoring Aug 17, 2026, 04:56 PM UTC

    We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery.

  4. monitoring Aug 17, 2026, 05:52 PM UTC

    We are still seeing intermitting GitHub API errors at a low rate.

  5. monitoring Aug 17, 2026, 07:32 PM UTC

    We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage.

  6. resolved Aug 17, 2026, 08:40 PM UTC

    This incident has been resolved.