Blacksmith incident

Job failures in us-west

Minor Resolved

Blacksmith experienced a minor incident on September 28, 2026 affecting US West (us-west ARM) and US West (us-west x86), lasting 1h 22m. The incident has been resolved; the full update timeline is below.

Started
Sep 28, 2026, 08:52 PM UTC
Resolved
Sep 28, 2026, 10:14 PM UTC
Duration
1h 22m
Detected by Pingoru
Sep 28, 2026, 08:52 PM UTC

Affected components

US West (us-west ARM)US West (us-west x86)

Update timeline

  1. identified Sep 28, 2026, 08:52 PM UTC

    A rack-level power failure at our us-west data center caused some jobs to fail with runner communication errors. The issue has been communicated with our upstream provider and we are identifying the affected machines to mitigate the impact to our customers.

  2. monitoring Sep 28, 2026, 09:53 PM UTC

    The affected rack remains offline while our upstream provider works to restore its power, and jobs in us-west are running normally on the rest of the fleet. Any jobs that failed with runner communication errors can be safely re-run.

  3. resolved Sep 28, 2026, 10:14 PM UTC

    This incident has been resolved.