Blacksmith incident

Service degradation in upstream GHCR registries

Minor Resolved

Blacksmith experienced a minor incident on September 2, 2026 affecting Github → API Requests, lasting 2h 21m. The incident has been resolved; the full update timeline is below.

Started
Sep 02, 2026, 02:09 PM UTC
Resolved
Sep 02, 2026, 04:30 PM UTC
Duration
2h 21m
Detected by Pingoru
Sep 02, 2026, 02:09 PM UTC

Affected components

Github → API Requests

Update timeline

  1. investigating Sep 02, 2026, 02:09 PM UTC

    We are currently observing higher rates of errors for the upstream GHCR registries which may indicate an undeclared GitHub incident. We are currently monitoring.

  2. investigating Sep 02, 2026, 02:20 PM UTC

    We are seeing evidence that Git Checkouts are also affected by this. We are seeing high TCP retransmit rates into the EU GitHub loadbalancer. Checkouts in the EU West, EU Central, US East regions are affected.

  3. monitoring Sep 02, 2026, 03:43 PM UTC

    Connections to [github.com](http://github.com) and [ghcr.io](http://ghcr.io) from our eu-west, eu-central, and us-east regions have returned to normal as of approximately 15:00 UTC, so checkouts are completing at normal speed and login failures to GitHub Container Registry have dropped to baseline levels. We are continuing to monitor, and re-running any jobs that failed during this period should succeed.

  4. resolved Sep 02, 2026, 04:30 PM UTC

    This incident has been resolved. GitHub connectivity from our EU regions was degraded from \~12:45 to 16:15 UTC, affecting checkouts, [ghcr.io](http://ghcr.io), and some eu-west job starts. Normal since 16:15\. Affected jobs can be re-run.