Blacksmith incident
Storage degradation in us-west
Blacksmith experienced a minor incident on August 13, 2026 affecting EU Central (eu-central ARM) and EU Central (eu-central x86) and 1 more component, lasting 1d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 13, 2026, 11:40 AM UTC
Our cloud provider is experiencing a thermal event in their us-west datacenter that is affecting caching in the region. Sticky Disk, Actions Cache, Docker Container Cache, and Bazel Build Caching are impacted: cache operations may be slow or unavailable, so some builds may run slower than usual.
- identified Aug 13, 2026, 12:06 PM UTC
Caching performance in US West is currently degraded. Our upstream provider has lost cooling power in their data center, leading to failure of some storage systems in US West. As a result jobs may take longer to complete. We are taking action to mitigate the issue.
- identified Aug 13, 2026, 12:36 PM UTC
We are currently applying mitigations for affected customers in the region.
- identified Aug 13, 2026, 01:02 PM UTC
Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions.
- identified Aug 13, 2026, 01:27 PM UTC
As we rebalance customers away from us-west, other regions may see slightly longer pickup times than usual as a side effect
- identified Aug 13, 2026, 01:59 PM UTC
Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions.
- identified Aug 13, 2026, 02:40 PM UTC
Conditions at our provider's us-west facility have not yet improved. We are continuing to issue mitigations to ensure that customer impact is minimal.
- identified Aug 13, 2026, 03:11 PM UTC
Conditions in our us-west region have not yet improved. We're actively working with our datacenter provider on the issue, and will provide updates as they occur. We are also manually rebalancing traffic out of the us-west region to aid with recovery. Github have also declared an incident affecting webhooks - which may cause some job adoption delays
- identified Aug 13, 2026, 04:32 PM UTC
Our datacenter provider has begun deploying mitigations at the affected us-west facility, and GitHub has resolved its separate webhooks incident, though some knock-on delays may persist while affected jobs requeue. The us-west region remains heavily affected and with capacity reduced while we rebalance traffic, jobs in all regions may take longer than usual to start.
- identified Aug 13, 2026, 05:00 PM UTC
Our provider has partially restored cooling at the us-west facility and expects to reach safe operating temperatures within the next few hours, at which point servers will be powered back on in phases. In the meantime jobs in all regions may still take significantly longer than usual to start. We will provide another update within the next hour.
- identified Aug 13, 2026, 05:42 PM UTC
Our provider is continuing to restore cooling at the us-west facility; temperatures have not yet reached safe levels for servers to be powered back on, and we are staged to begin restoring runner capacity as soon as they are. The us-west region remains in a major outage, and jobs in all regions may still take longer than usual to start. We will provide another update within the next hour.
- identified Aug 13, 2026, 06:26 PM UTC
Our provider has begun powering servers back on at the us-west facility, and a portion of our us-west capacity is back online. We have started moving some customers back to us-west to spread load across regions. Jobs in all regions may still take longer than usual to start.
- identified Aug 13, 2026, 07:15 PM UTC
Our provider is continuing to bring us-west servers back online, and we are gradually moving customers back to us-west as capacity returns. The backlog has not yet cleared, so jobs in all regions may still take longer than usual to start. We will provide another update within the next hour.
- identified Aug 13, 2026, 07:59 PM UTC
More us-west capacity has come back online and queues in the region are steadily draining as we move customers back. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage portion of the facility, and because we shifted us-west traffic to other regions earlier today, jobs elsewhere, particularly in us-east, may still take longer than usual to start. We will provide another update within the next hour.
- identified Aug 13, 2026, 08:44 PM UTC
Queues in us-west have come down substantially as restored capacity comes online and we continue moving customers back. Cache operations, including sticky disks, remain unavailable in us-west, and jobs in us-east may still take significantly longer than usual to start while we work through the remaining backlog. We will provide another update within the next hour.
- identified Aug 13, 2026, 09:48 PM UTC
Queue times in us-west have returned to near-normal levels, while us-east, eu-west, and eu-central are still working through their remaining backlogs. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage systems there. We will provide another update within the next hour.
- identified Aug 13, 2026, 10:54 PM UTC
Queue times are improving across all regions as capacity comes back online, but jobs may still take longer than usual to start. Cache operations in us-west, including sticky disks, remain unavailable while our provider restores the storage systems, and jobs on our largest runner sizes may see the longest delays. We will provide another update within the next hour.
- identified Aug 14, 2026, 12:00 AM UTC
Runner capacity in us-west continues to recover and queues in all regions are draining. Our provider has restored the access we need to begin bringing our us-west storage systems back online. Cache operations, including sticky disks, remain unavailable while that work completes, and jobs may still take longer than usual to start.
- monitoring Aug 14, 2026, 01:15 AM UTC
Queue times for all runner sizes have returned to normal in all regions, and we are monitoring closely while eu-west clears the last of its backlog. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour.
- monitoring Aug 14, 2026, 02:08 AM UTC
Job queues have fully recovered, and runners in all regions are operating normally for all runner sizes. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour.
- monitoring Aug 14, 2026, 03:06 AM UTC
Our provider has restored power and cooling in us-west, our capacity has returned. Cache operations, including sticky disks, remain unavailable in us-west while we complete storage recovery, and we will post an update when they are restored.
- monitoring Aug 14, 2026, 03:26 AM UTC
We are noticing some issues with the storage cluster backing the Github Actions cache after it was restored. We are working on restoring this alongside the storage cluster backing Sticky Disks, Incremental Docker Builders, and Docker Container Caching.
- monitoring Aug 14, 2026, 04:09 AM UTC
We are still working with our compute provider to recover the storage clusters.
- monitoring Aug 14, 2026, 06:06 AM UTC
We have restored the Github Actions Cache in us-west. We are still working with our compute provider to restore our remaining storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders
- monitoring Aug 14, 2026, 10:57 AM UTC
We are continuing to work with our compute provider to restore the storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west. This requires hands-on recovery work by our provider's team and may take a few more hours to fully restore. Workflows using the affected features in us-west may see slow or stalled Docker builds in the meantime.
- monitoring Aug 14, 2026, 03:02 PM UTC
The storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west has been restored, and disks are mounting and operating normally. As part of the recovery, a small amount of recently written cache data may need to be rebuilt, so some Docker builds may run slower over their first few runs while caches rehydrate. We are monitoring closely.
- monitoring Aug 14, 2026, 04:16 PM UTC
All services have been restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. We are continuing to monitor the stability of the recovered storage cluster as it ramps back up with traffic, and some builds may run slower on their first runs while recently written cache data rebuilds. We will post a final update once we have confirmed stability.
- resolved Aug 14, 2026, 06:14 PM UTC
All services have been fully restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. A small amount of recently written cache data could not be recovered during storage repair, so some builds may run slower on their first runs while caches rebuild. This incident is now resolved, and we will publish a detailed post-incident report in the coming days.