Blacksmith Outage History
Blacksmith is up right nowPingoru has recorded 103 incidents for Blacksmith: 103 in the last 12 months and 27 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 10 | 19 of 720 (3%) |
| May 2026 | 15 | 207 of 744 (28%) |
| June 2026 | 12 | 18 of 720 (2%) |
| July 2026 | 16 | 45 of 744 (6%) |
| August 2026 | 13 | 70 of 744 (9%) |
| September 2026 | 11 | 24 of 720 (3%) |
| October 2026 | 2 | 15 of 574 (3%) |
Incidents, newest first
Minor August 2, 2026 US-West storage cluster degraded 20m Incremental Docker Builders (us-west Storage Cluster), Docker Container Cache (us-west Storage Cluster), Sticky Disks (us-west Storage Cluster)
Started Aug 02, 2026, 06:16 AM UTC· resolved Aug 02, 2026, 06:37 AM UTC
- investigating Aug 02, 2026, 06:16 AM UTC
We are seeing some failures with sticky disk availability and write latency with the US-West storage cluster.
- monitoring Aug 02, 2026, 06:25 AM UTC
We've applied a fix and are seeing failure rates and latency starting to come down.
- resolved Aug 02, 2026, 06:37 AM UTC
This incident has been resolved.
Major July 29, 2026 Increased Error Rate on Codesmith Sandbox Startup 26m Codesmith
Started Jul 29, 2026, 04:05 PM UTC· resolved Jul 29, 2026, 04:31 PM UTC
- identified Jul 29, 2026, 04:05 PM UTC
We're seeing increased error rates from an upstream provider in spinning up sandbox environments in Codesmith. We are looking into remediation options.
- monitoring Jul 29, 2026, 04:08 PM UTC
The upstream issue is resolved, we are monitoring the sandbox creation rate recovering.
- resolved Jul 29, 2026, 04:31 PM UTC
This incident has been resolved.
Minor July 24, 2026 Jobs not being adopted in all regions 33m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 3 more
Started Jul 24, 2026, 07:17 PM UTC· resolved Jul 24, 2026, 07:51 PM UTC
- investigating Jul 24, 2026, 07:17 PM UTC
We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating.
- monitoring Jul 24, 2026, 07:20 PM UTC
We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes.
- monitoring Jul 24, 2026, 07:23 PM UTC
We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly.
- resolved Jul 24, 2026, 07:51 PM UTC
This incident has been resolved, and job pickup has returned to normal.
Minor July 23, 2026 Sticky disk degradation in EU-West 43m Sticky Disks (eu-west Storage Cluster), Docker Container Cache (eu-west Storage Cluster)
Started Jul 23, 2026, 03:00 PM UTC· resolved Jul 23, 2026, 03:44 PM UTC
- investigating Jul 23, 2026, 03:00 PM UTC
We're seeing large volumes of traffic in our storage cluster in EU-West and this is causing a number of sticky disk related requests to time out and are actively investigating.
- monitoring Jul 23, 2026, 03:18 PM UTC
We implemented a fix and are currently monitoring the result.
- resolved Jul 23, 2026, 03:44 PM UTC
This incident has been resolved.
Minor July 22, 2026 Extended Queueing in Action Runners 20m US West (us-west ARM), US West (us-west x86), EU West (eu-west x86)
Started Jul 22, 2026, 05:45 PM UTC· resolved Jul 22, 2026, 06:05 PM UTC
- investigating Jul 22, 2026, 05:45 PM UTC
We are currently experiencing elevated tail latencies for certain customers due to job prioritization. We are noticing jobs are taking upwards of 10m to adopt in a few cases. We are looking at options to alleviate queueing.
- resolved Jul 22, 2026, 06:05 PM UTC
We have implemented some mitigations and queue times are back to normal. The incident has been resolved.
Minor July 22, 2026 Degredated actions cache performance 1h 14m Actions Cache
Started Jul 22, 2026, 01:00 PM UTC· resolved Jul 22, 2026, 02:14 PM UTC
- monitoring Jul 22, 2026, 01:00 PM UTC
We identified an issue which was leading to actions/cache requests experiencing an elevated failure rate. This has now been mitigated and we are monitoring.
- monitoring Jul 22, 2026, 01:18 PM UTC
We implemented a fix and are currently monitoring the result.
- resolved Jul 22, 2026, 02:14 PM UTC
This incident has been resolved.
Major July 21, 2026 Delays in job adoption 9h 5m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 12 more
Started Jul 21, 2026, 02:25 PM UTC· resolved Jul 21, 2026, 11:31 PM UTC
- investigating Jul 21, 2026, 02:25 PM UTC
We are currently investigating an issue which is affecting Caching and Monitors for some customers.
- identified Jul 21, 2026, 03:44 PM UTC
We are continuing to investigate the delays in webhook processing on our control plane. Customer's may also see some degradation with caching and sticky disk components.
- identified Jul 21, 2026, 05:49 PM UTC
We're still seeing large scale degradation over the backend. likely most caching + storage agent requests are still failing which is causing to queueing due to the slowdown in builds. Actively investigating and working on mitigations.
- investigating Jul 21, 2026, 06:39 PM UTC
We have rolled out fixes to our workload to bring database query latency back to baseline. The degradation has now moved to other parts of our stack. We are continuing to investigate the root cause here. Customers can still expect to see degraded cache interactions. We will provide an update within the next 30 minutes.
- investigating Jul 21, 2026, 07:03 PM UTC
We have applied a change to our backend systems and metrics are showing partial improvement, though not yet back to baseline. Customers can still expect degraded cache interactions while we continue to investigate the root cause. We will provide an update within the next 30 minutes.
- monitoring Jul 21, 2026, 07:38 PM UTC
Our primary Redis instance, which backs our control plane, hit a saturation point, leading to a feedback loop of load. The initial cause was a burst of deliveries of delayed webhooks from GitHub, and our reconciliation systems added further load to the control plane, making things worse. We have improved load balancing by spreading this workload across multiple Redis instances, and our control plane has fully recovered. The remaining impact is a backlog of queued jobs that we are actively draining, so some customers may still see delayed job starts until the queue clears. We will continue monitoring and update with our findings in the next 30 minutes.
- monitoring Jul 21, 2026, 07:56 PM UTC
Cache and sticky disk operations are healthy again. Bazel caching is not yet operational but being actively investigated by our team. The remaining impact is a backlog of queued jobs that we are actively draining, so some customers may still see delayed job starts until the queue drains.
- monitoring Jul 21, 2026, 09:25 PM UTC
Job in us-west and eu-west have returned to normal. Customers in eu-central may still see delayed job starts while we clear the remaining queue backlog. We have identified the cause of the Bazel caching issue and are now implementing a fix. We will provide another update within the next hour.
- monitoring Jul 21, 2026, 10:30 PM UTC
We are still watching over the job queue draining in one of our regions (eu-central). Customers in this region will see a delay for their jobs to be picked up.
- monitoring Jul 21, 2026, 10:59 PM UTC
We are seeing job adoption return to normal latencies across all regions.
- monitoring Jul 21, 2026, 11:19 PM UTC
We are currently scanning for any missed jobs during this degradation period and ensuring that they are being dispatched.
- resolved Jul 21, 2026, 11:31 PM UTC
This incident has been resolved.
Major July 19, 2026 GitHub Outage Affecting Jobs 4h 15m Github → Actions
Started Jul 19, 2026, 11:38 PM UTC· resolved Jul 20, 2026, 03:54 AM UTC
- identified Jul 19, 2026, 11:38 PM UTC
GitHub has opened an incident which we are seeing is affecting jobs getting picked up.
- identified Jul 20, 2026, 12:28 AM UTC
We are continuing to see errors from GitHub when trying to provision CI jobs. They are continuing to investigate their incident.
- identified Jul 20, 2026, 02:21 AM UTC
We are starting to see signs of recovery of upstream API errors and jobs have resumed running. We are continuing to monitor.
- identified Jul 20, 2026, 03:08 AM UTC
Requests to GitHub API are now succeeding. There maybe some some delays in job adoption as we work through the large backlog of jobs that did not run during the incident.
- monitoring Jul 20, 2026, 03:35 AM UTC
Job executions have returned to normal. We are monitoring the recovery.
- resolved Jul 20, 2026, 03:54 AM UTC
This incident has been resolved.
Minor July 16, 2026 Intermittent dashboard loading and CI job errors due to GitHub outage 1h 29m Website
Started Jul 16, 2026, 10:34 PM UTC· resolved Jul 17, 2026, 12:03 AM UTC
- investigating Jul 16, 2026, 10:34 PM UTC
We are currently investigating this incident.
- identified Jul 16, 2026, 10:36 PM UTC
We are seeing errors affecting the availability to log into the Blacksmith dashboard due to upstream calls to GitHub. We are not currently seeing any impact to the execution of CI jobs.
- monitoring Jul 16, 2026, 10:55 PM UTC
We are seeing errors affecting the availability to log into the Blacksmith dashboard due to upstream calls to GitHub. There are widespread reports of GitHub API failures: We are seeing this impact CI jobs that interact with GitHub APIs such as downloading actions, uploading artifacts, etc.
- resolved Jul 17, 2026, 12:03 AM UTC
The upstream GitHub incident has been resolved: We will continue to monitor GitHub API Stability.
Minor July 9, 2026 Sticky disk attach failures in us-west 12m Sticky Disks (us-west Storage Cluster)
Started Jul 09, 2026, 09:09 PM UTC· resolved Jul 09, 2026, 09:21 PM UTC
- investigating Jul 09, 2026, 09:09 PM UTC
We are currently experiencing an issue in our us-west region where sticky disks are failing to attach. Sticky disks back several of our caching features, so customers running jobs in us-west may be affected across Docker container caching, Docker build caching, and Git repository caching. Affected jobs will still run, but will fall back to uncached behavior and may take longer than usual. Runners in other regions are unaffected. We are actively investigating and will post updates as we make progress.
- resolved Jul 09, 2026, 09:21 PM UTC
This incident is resolved. The issue was identified and fixed. Sticky disk attachments in us-west were affected between 20:40 and 21:15 UTC.
Minor July 7, 2026 Github Actions and Codespaces APIs experiencing partial failures 1h 55m Github → Actions, Github → API Requests
Started Jul 07, 2026, 02:36 PM UTC· resolved Jul 07, 2026, 04:31 PM UTC
- investigating Jul 07, 2026, 02:36 PM UTC
Github is investigating reports of degraded performance for Actions and Codespaces.
- resolved Jul 07, 2026, 04:31 PM UTC
This incident has been resolved.
Minor July 6, 2026 Delays in job adoption in us-west 1h 53m US West (us-west x86), Incremental Docker Builders (us-west Storage Cluster), Docker Container Cache (us-west Storage Cluster)
Started Jul 06, 2026, 08:25 PM UTC· resolved Jul 06, 2026, 10:18 PM UTC
- investigating Jul 06, 2026, 08:25 PM UTC
We're seeing a spike of jobs in the us-west region leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delaysWe are currently investigating this incident.
- resolved Jul 06, 2026, 10:18 PM UTC
This incident has been resolved.
Minor July 6, 2026 Actions Cache degradation in us-west 6h 31m Actions Cache, Incremental Docker Builders (us-west Storage Cluster), Docker Container Cache (us-west Storage Cluster)
Started Jul 06, 2026, 03:23 PM UTC· resolved Jul 06, 2026, 09:55 PM UTC
- investigating Jul 06, 2026, 03:23 PM UTC
We are currently investigating this incident.
- investigating Jul 06, 2026, 03:40 PM UTC
We are continuing to investigate a network issue with a storage server in our US West region since around 14:50 UTC on July 6\. Customers may experience slowness in US West. We are working with our upstream network provider on a fix and will post updates here.
- identified Jul 06, 2026, 07:21 PM UTC
We are running a maintenance operation on our cache storage cluster to resolve the degradation. For roughly 15 to 20 minutes, caching will fall back to the GitHub Actions cache and may be slower; jobs will otherwise run normally. We will update once complete.
- monitoring Jul 06, 2026, 09:08 PM UTC
We implemented a fix and are currently monitoring the result.
- resolved Jul 06, 2026, 09:55 PM UTC
This incident has been resolved.
Minor July 2, 2026 Missing job history in the Blacksmith dashboard for runs before 3:30pm EDT today 13h 57m API
Started Jul 02, 2026, 09:03 PM UTC· resolved Jul 03, 2026, 11:01 AM UTC
- identified Jul 02, 2026, 09:03 PM UTC
All job history rows from before 3:30pm EDT today were lost. Data up is being restored from a backup.
- monitoring Jul 03, 2026, 12:41 AM UTC
We are in the process of backfilling the data and customers should begin to start seeing data from 12am and earlier on their observability pages. We are still working on a resolution for today's gap (12AM - 3:30PM) and will update as soon as that fix is put in place.
- resolved Jul 03, 2026, 11:01 AM UTC
This incident has been resolved. We will continue to backfill certain impacted time periods but observability data for customers is now accurately represented in the workflow and job dashboards.
Major July 2, 2026 Actions Cache degradation 2h 58m Actions Cache
Started Jul 02, 2026, 01:17 PM UTC· resolved Jul 02, 2026, 04:16 PM UTC
- investigating Jul 02, 2026, 01:17 PM UTC
We are seeing some instances of actions cache degradation post incident are are actively investigating.
- monitoring Jul 02, 2026, 02:49 PM UTC
We implemented a fix and are currently monitoring the result.
- resolved Jul 02, 2026, 04:16 PM UTC
This incident has been resolved.
Major July 2, 2026 Actions Cache outage 1h 11m Actions Cache
Started Jul 02, 2026, 02:52 AM UTC· resolved Jul 02, 2026, 04:03 AM UTC
- investigating Jul 02, 2026, 02:52 AM UTC
We are currently investigating this incident.
- monitoring Jul 02, 2026, 03:38 AM UTC
We implemented a fix and are currently monitoring the result.
- resolved Jul 02, 2026, 04:03 AM UTC
This incident has been resolved.
Major June 30, 2026 Runners hitting Github rate limiting 1h 11m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 4 more
Started Jun 30, 2026, 05:36 PM UTC· resolved Jun 30, 2026, 06:47 PM UTC
- investigating Jun 30, 2026, 05:36 PM UTC
We are currently investigating an issue where customers Github Actions are not being picked up by Runners across our regions.
- identified Jun 30, 2026, 06:28 PM UTC
We are have mitigated the issue on our end, but customers may experience some queueing until we catch up.
- monitoring Jun 30, 2026, 06:35 PM UTC
Queueing improving and currently monitoring.
- resolved Jun 30, 2026, 06:47 PM UTC
This incident has been resolved.
Minor June 27, 2026 EU Central Storage Cluster 54m Incremental Docker Builders (eu-central Storage Cluster), Docker Container Cache (eu-central Storage Cluster), Sticky Disks (eu-central Storage Cluster)
Started Jun 27, 2026, 09:21 PM UTC· resolved Jun 27, 2026, 10:16 PM UTC
- investigating Jun 27, 2026, 09:21 PM UTC
We are currently investigating some service degradation with the eu-central storage cluster. You may see cache misses and slower disk speeds.
- monitoring Jun 27, 2026, 09:51 PM UTC
We implemented a fix and are currently monitoring the result.
- resolved Jun 27, 2026, 10:16 PM UTC
This incident has been resolved.
Minor June 26, 2026 Delays in MacOS job adoption 53m US Central (us-central MacOS)
Started Jun 26, 2026, 12:21 PM UTC· resolved Jun 26, 2026, 01:15 PM UTC
- investigating Jun 26, 2026, 12:21 PM UTC
We're seeing a spike of MacOS jobs leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays
- monitoring Jun 26, 2026, 12:50 PM UTC
Our team have mitigated this and are now seeing job pick up times return to normal as we monitor
- resolved Jun 26, 2026, 01:15 PM UTC
This incident has been resolved.
Minor June 25, 2026 Github are reporting degraded performance for Actions and Webhooks 7m EU Central (eu-central x86), US West (us-west x86), EU West (eu-west x86)
Started Jun 25, 2026, 06:29 PM UTC· resolved Jun 25, 2026, 06:36 PM UTC
- identified Jun 25, 2026, 06:29 PM UTC
Github has reported degraded performance for Actions, Pull Requests and Webhooks. Job may take a moment to be adopted. We are monitoring this incident.
- resolved Jun 25, 2026, 06:36 PM UTC
This incident has been resolved.
Major June 17, 2026 Job adoption delays 1h 3m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 4 more
Started Jun 17, 2026, 12:45 PM UTC· resolved Jun 17, 2026, 01:48 PM UTC
- investigating Jun 17, 2026, 12:45 PM UTC
We are receiving reports of job adoption delays, we are currently investigating.
- investigating Jun 17, 2026, 01:01 PM UTC
We believe this is related to an issue with Github Webhooks and are still investigating.
- identified Jun 17, 2026, 01:14 PM UTC
We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.
- monitoring Jun 17, 2026, 01:27 PM UTC
Our engineers are implementing a mitigation which should take 20 minutes to deploy. Thank you for your patience.
- monitoring Jun 17, 2026, 01:48 PM UTC
We implemented a fix and are currently monitoring the result but seeing recovery. Any existing queued jobs wont self recover so please be sure to cancel and rerun those jobs.
- resolved Jun 17, 2026, 01:48 PM UTC
This incident has been resolved.
Major June 15, 2026 Github are reporting degraded performance for Webhooks 59m Github → Webhooks
Started Jun 15, 2026, 03:45 PM UTC· resolved Jun 15, 2026, 04:44 PM UTC
- monitoring Jun 15, 2026, 03:45 PM UTC
GitHub are reporting degraded performance with Webhooks. This may have an impact on job adoption. We are monitoring this incident -
- resolved Jun 15, 2026, 04:44 PM UTC
Github have reported this incident as resolved.
Minor June 11, 2026 Currently investigating our degraded us-west storage cluster 2h 8m Incremental Docker Builders (us-west Storage Cluster), Docker Container Cache (us-west Storage Cluster), Sticky Disks (us-west Storage Cluster)
Started Jun 11, 2026, 08:54 PM UTC· resolved Jun 11, 2026, 11:02 PM UTC
- investigating Jun 11, 2026, 08:54 PM UTC
We are currently investigating this incident.
- resolved Jun 11, 2026, 11:02 PM UTC
Speeds have returned to normal and this incident has been resolved.
Major June 11, 2026 Delays In Job Adoption 3h 2m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 2 more
Started Jun 11, 2026, 04:35 PM UTC· resolved Jun 11, 2026, 07:38 PM UTC
- investigating Jun 11, 2026, 04:35 PM UTC
We're seeing a spike in jobs in some regions leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays
- identified Jun 11, 2026, 05:19 PM UTC
We are continuing to work on a fix for this incident. eu-central and us-west still experiencing some queueing, and eu-west has recovered.
- monitoring Jun 11, 2026, 06:41 PM UTC
us-west still experiencing some mild queueing, and eu-central and eu-west has recovered.
- resolved Jun 11, 2026, 07:38 PM UTC
This incident has been resolved.
Major June 10, 2026 Github reporting API request degradation 1h 23m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 3 more
Started Jun 10, 2026, 03:29 PM UTC· resolved Jun 10, 2026, 04:52 PM UTC
- investigating Jun 10, 2026, 03:29 PM UTC
We are currently investigating some reports of delays with job adoption. We are actively looking into this.
- monitoring Jun 10, 2026, 04:06 PM UTC
We are currently in the process of re-queuing jobs
- resolved Jun 10, 2026, 04:52 PM UTC
Github have reported this incident as resolved