Blacksmith experienced a minor incident on July 21, 2026, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jul 21, 2026, 02:25 PM UTC
Type: Incident Duration: 9 hours and 6 minutes Affected Components: eu-west Storage Cluster, eu-west Storage Cluster, eu-west Storage Cluster, eu-west x86, eu-central x86, us-west ARM, us-west x86, eu-central Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster, us-central MacOS, eu-central ARM, us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster Jul 21, 17:49:19 GMT+0 - Identified - We are continuing to investigate degradation in our control plane services. This is causing a large % of caching and stickydisk requests to fail, which in turn cause customer jobs to run much slower than their baseline. The slower runs are exacerbating queueing of new jobs that are coming in to the system. We are actively investigating and rolling out fixes to unblock customer jobs. Jul 21, 18:39:36 GMT+0 - Investigating - We have rolled out fixes to our workload to bring database query latency back to baseline. The degradation has now moved to other parts of our stack. We are continuing to investigate the root cause here. Customers can still expect to see degraded cache interactions. We will provide an update within the next 30 minutes. Jul 21, 19:03:57 GMT+0 - Investigating - We have applied a change to our backend systems and metrics are showing partial improvement, though not yet back to baseline. Customers can still expect degraded cache interactions while we continue to investigate the root cause. We will provide an update within the next 30 minutes. Jul 21, 19:38:23 GMT+0 - Monitoring - Our primary Redis instance, which backs our control plane, hit a saturation point, leading to a feedback loop of load. The initial cause was a burst of deliveries of delayed webhooks from GitHub, and our reconciliation systems added further load to the control plane, making things worse. We have improved load balancing by spreading this workload across multiple Redis instances, and our control plane has fully recovered. The remaining impact is a backlog of queued jobs that we are actively draining, so some customers may still see delayed job starts until the queue clears. We will continue monitoring and update with our findings in the next 30 minutes. Jul 21, 19:56:41 GMT+0 - Monitoring - Cache and sticky disk operations are healthy again. Bazel caching is not yet operational but being actively investigated by our team. The remaining impact is a backlog of queued jobs that we are actively draining, so some customers may still see delayed job starts until the queue drains. Jul 21, 21:25:15 GMT+0 - Monitoring - Job adoption in us-west and eu-west have returned to normal. Customers in eu-central may still see delayed job starts while we clear the remaining queue backlog. We have identified the cause of the Bazel caching issue and are now implementing a fix. We will provide another update within the next hour. Jul 21, 22:30:44 GMT+0 - Monitoring - We are still watching over the job queue draining in one of our regions (eu-central). Customers in this region will see a delay for their jobs to be picked up. Jul 21, 22:59:51 GMT+0 - Monitoring - We are seeing job adoption return to normal latencies across all regions. Jul 21, 23:19:55 GMT+0 - Monitoring - We are currently scanning for any missed jobs during this degradation period and ensuring that they are being dispatched. Jul 21, 23:31:33 GMT+0 - Resolved - This incident has been resolved. Jul 21, 14:25:57 GMT+0 - Identified - We have applied a mitigation for the Caching and Monitors degradation. We are seeing some delays in webhook processing on our control plane which we are investigating. Jul 21, 15:44:14 GMT+0 - Identified - We are continuing to investigate the delays in webhook processing on our control plane. Customer's may also see some degradation with caching and sticky disk components.