Blacksmith Outage History

Blacksmith is up right now

Pingoru has recorded 103 incidents for Blacksmith: 103 in the last 12 months and 27 in the last 90 days.

Source: status.blacksmith.sh

103incidents recorded
103last 12 months
27last 90 days
4 hourstypical duration
Incidents per month
01020AprMayJunJulAugSepOct2
Share of hours with an open incident, any severity
0%50%100%AprMayJunJulAugSepOct3%
Pingoru began tracking Blacksmith in April 2026, so earlier months are not shown. Hover a bar for its value.
Show as a table
MonthIncidentsHours with an open incident
April 20261019 of 720 (3%)
May 202615207 of 744 (28%)
June 20261218 of 720 (2%)
July 20261645 of 744 (6%)
August 20261370 of 744 (9%)
September 20261124 of 720 (3%)
October 2026215 of 582 (3%)

Incidents, newest first

Minor June 8, 2026 Oauth Issues Accessing Blacksmith Dashboard 2h 53m Website

Started Jun 08, 2026, 01:44 PM UTC· resolved Jun 08, 2026, 04:37 PM UTC

  1. investigating Jun 08, 2026, 01:44 PM UTC

    We are currently investigating an issue affecting some customers ability to log into the dashboard

  2. monitoring Jun 08, 2026, 03:12 PM UTC

    We now see users logging into the dashboard. The issue appears to be upstream and we are continuing to monitor.

  3. resolved Jun 08, 2026, 04:37 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor June 8, 2026 GitHub reporting degraded performance for Actions 8m Github → Actions

Started Jun 08, 2026, 08:34 AM UTC· resolved Jun 08, 2026, 08:43 AM UTC

  1. monitoring Jun 08, 2026, 08:34 AM UTC

    GitHub is reporting degraded performance with Actions as well as other services. This may have an impact on job adoption. We are monitoring this incident -

  2. resolved Jun 08, 2026, 08:43 AM UTC

    GitHub have resolved this incident.

Read the full incident report →

Major May 27, 2026 Delay in picking up jobs 5h 16m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 2 more

Started May 27, 2026, 07:01 PM UTC· resolved May 28, 2026, 12:18 AM UTC

  1. investigating May 27, 2026, 07:01 PM UTC

    We are currently investigating this incident.

  2. identified May 27, 2026, 07:56 PM UTC

    We've deployed mitigations and are observing improvements.

  3. monitoring May 27, 2026, 08:24 PM UTC

    Continuing to monitor improvements from deployment.

  4. monitoring May 27, 2026, 10:35 PM UTC

    We're continuing to monitor the situation. There's still a slight delay in jobs, but cancelling and rerunning the jobs should resolve the issue.

  5. resolved May 28, 2026, 12:18 AM UTC

    This incident has been resolved.

Read the full incident report →

Major May 27, 2026 Github reporting degraded performance for API Requests, Git Operations and other services 32m Github → API Requests

Started May 27, 2026, 12:53 PM UTC· resolved May 27, 2026, 01:26 PM UTC

  1. monitoring May 27, 2026, 12:53 PM UTC

    We will continue to monitor -

  2. resolved May 27, 2026, 01:26 PM UTC

    Github have resolved this incident.

Read the full incident report →

Minor May 20, 2026 Delays in job adoption in eu-west 2h 39m EU West (eu-west x86)

Started May 20, 2026, 04:33 PM UTC· resolved May 20, 2026, 07:13 PM UTC

  1. investigating May 20, 2026, 04:33 PM UTC

    We're seeing a spike of jobs in the eu-west region leading to a temporary capacity shortage. We're looking into re-balancing to reduce the delays

  2. identified May 20, 2026, 05:30 PM UTC

    We are continuing to monitor this, as well as track an incident reported by with GitHub Actions which can be found here: We are continuing to re-balance in order to reduce delays.

  3. resolved May 20, 2026, 07:13 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor May 20, 2026 Transparent caching degradaded in eu-west 7d 2h Actions Cache

Started May 20, 2026, 01:01 PM UTC· resolved May 27, 2026, 03:25 PM UTC

  1. monitoring May 20, 2026, 01:01 PM UTC

    We noticed a small uptick in transparent caching failures in our eu-west region, and have applied a potential fix.

  2. identified May 20, 2026, 02:41 PM UTC

    We have identified a bad host in the fleet and are working on replacing it. Client cache interactions are recovering as we rollout this fix.

  3. identified May 20, 2026, 10:59 PM UTC

    We're currently adding more capacity to the cluster. There will be a few minutes of slow cache interactions while the operation completes.

  4. monitoring May 21, 2026, 12:00 AM UTC

    The cluster expansion is complete. Our testing shows that cache interactions are no longer degraded. We're monitoring the cluster to confirm complete recovery.

Read the full incident report →

Minor May 19, 2026 GitHub webhooks degraded causing job queueing 8h 49m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 2 more

Started May 19, 2026, 02:50 PM UTC· resolved May 19, 2026, 11:39 PM UTC

  1. investigating May 19, 2026, 02:50 PM UTC

    We're seeing evidence of webhooks delivery being degraded from GitHub. We're investigating.

  2. monitoring May 19, 2026, 03:53 PM UTC

    We're seeing upstream recovery for GitHub webhook deliveries. Jobs may queue as we process GitHub's backlog of webhook events.

  3. monitoring May 19, 2026, 05:10 PM UTC

    We're still seeing a large backlog of queued jobs due to the incident that the system is working through, we're exploring mitigations.

  4. monitoring May 19, 2026, 08:02 PM UTC

    We're still working through a substantial backlog of queued tasks that has accumulated over this period of delayed webhook arrivals.

  5. monitoring May 19, 2026, 09:22 PM UTC

    We are close to the end of the backlog of the queued tasks and are seeing full recovery in certain runner pools.

  6. resolved May 19, 2026, 11:39 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor May 15, 2026 Github Actions degraded 9m Github → Actions

Started May 15, 2026, 08:20 AM UTC· resolved May 15, 2026, 08:29 AM UTC

  1. identified May 15, 2026, 08:20 AM UTC

    GitHub is reporting a GitHub Actions degradation, and we're monitoring the impact -

  2. resolved May 15, 2026, 08:29 AM UTC

    Github have reportedly resolved the incident

Read the full incident report →

Minor May 14, 2026 Job Queueing due to Capacity Event 1h 9m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 2 more

Started May 14, 2026, 09:20 PM UTC· resolved May 14, 2026, 10:30 PM UTC

  1. investigating May 14, 2026, 09:20 PM UTC

    We are currently experiencing a high volume of jobs which is causing queuing in job adoption.

  2. resolved May 14, 2026, 10:30 PM UTC

    This incident has been resolved.

Read the full incident report →

Major May 12, 2026 Webhook Service Outage 3h 7m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 2 more

Started May 12, 2026, 08:39 PM UTC· resolved May 12, 2026, 11:46 PM UTC

  1. investigating May 12, 2026, 08:39 PM UTC

    We are currently investigating this incident.

  2. monitoring May 12, 2026, 08:42 PM UTC

    We implemented a fix and are currently monitoring the result.

  3. resolved May 12, 2026, 11:46 PM UTC

    We have been running into load issues in our database layer that's causing sporadic instability in job adoption times. We have been actively working with our database provider to improve our systems to handle the recently increased load. We are planning proactive maintenance later this week to address this further.

Read the full incident report →

Major May 11, 2026 Blacksmith control plane outage 6h 35m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 2 more

Started May 11, 2026, 03:56 PM UTC· resolved May 11, 2026, 10:31 PM UTC

  1. investigating May 11, 2026, 03:56 PM UTC

    There may be some delays with job adoption

  2. monitoring May 11, 2026, 03:58 PM UTC

    We implemented a fix and are re-queuing jobs in the backlog

  3. monitoring May 11, 2026, 07:32 PM UTC

    We are still investigating issues with slow job adoption.

  4. identified May 11, 2026, 08:30 PM UTC

    We are continuing to face issues with our upstream database provider. We are working with them to root cause the problem.

  5. resolved May 11, 2026, 10:31 PM UTC

    This incident has been resolved. Queue times are back to normal, we are working on mitigations on our end to prevent such load based degradations in the future.

Read the full incident report →

Minor May 9, 2026 Elevated queue time for mac runners 1h 39m US Central (us-central MacOS)

Started May 09, 2026, 08:58 PM UTC· resolved May 09, 2026, 10:37 PM UTC

  1. investigating May 09, 2026, 08:58 PM UTC

    We are currently investigating this incident.

  2. identified May 09, 2026, 09:38 PM UTC

    A fix for the issue has been deployed, we expect queue times to start returning to normal momentarily.

  3. resolved May 09, 2026, 10:37 PM UTC

    This incident has been resolved, mac queue times are back to normal.

Read the full incident report →

Major May 8, 2026 Elevated tail latency in git checkout operations 2h 8m US West (us-west ARM), US West (us-west x86)

Started May 08, 2026, 06:22 PM UTC· resolved May 08, 2026, 08:30 PM UTC

  1. identified May 08, 2026, 06:22 PM UTC

    We have been made aware of an upstream degradation in our US West region by our network provider. This is manifesting as some % of git checkout operations taking much longer than their baseline. We are working with our upstream provider to restore functionality.

  2. monitoring May 08, 2026, 07:54 PM UTC

    The upstream degradation has been resolved, we are seeing checkout times return to normal.

  3. resolved May 08, 2026, 08:30 PM UTC

    This incident has been resolved, checkout operations are back to their baseline.

Read the full incident report →

Minor May 4, 2026 GitHub Actions and Webhooks degradation 2h 9m Github → Actions, Github → Webhooks

Started May 04, 2026, 04:04 PM UTC· resolved May 04, 2026, 06:13 PM UTC

  1. monitoring May 04, 2026, 04:04 PM UTC

    GitHub is reporting a GitHub Actions and Webhooks degradation, and we're monitoring the impact

  2. monitoring May 04, 2026, 04:36 PM UTC

    GitHub are still experiencing issues with Webhooks

  3. resolved May 04, 2026, 06:13 PM UTC

    This incident has been resolved.

Read the full incident report →

Major April 30, 2026 Degraded Actions performance in us-west 4h 18m US West (us-west x86)

Started Apr 30, 2026, 01:17 PM UTC· resolved Apr 30, 2026, 05:36 PM UTC

  1. monitoring Apr 30, 2026, 01:17 PM UTC

    We are aware of some reports of slow Actions performance in our us-west region. A mitigation has been applied and we're now seeing signs of recovery

  2. monitoring Apr 30, 2026, 02:11 PM UTC

    We are continuing to see recovery in us-west

  3. resolved Apr 30, 2026, 05:36 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor April 28, 2026 Blacksmith control plane DB maintenance 2h 4m EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM), US West (us-west x86) and 2 more

Started Apr 28, 2026, 03:45 PM UTC· resolved Apr 28, 2026, 05:50 PM UTC

  1. investigating Apr 28, 2026, 03:45 PM UTC

    We are performing brief maintenance on our database. Customers may experience some delays with job adoption

  2. monitoring Apr 28, 2026, 03:47 PM UTC

    We have completed the DB operation and currently re-queuing jobs. This may take \~15 minutes

  3. monitoring Apr 28, 2026, 04:44 PM UTC

    The re-queuing is still in progress however new jobs are being picked up without delay.

  4. resolved Apr 28, 2026, 05:50 PM UTC

    This incident has been resolved.

Read the full incident report →

Major April 27, 2026 Github Actions Degraded 7h 22m Overall

Started Apr 27, 2026, 04:38 PM UTC· resolved Apr 28, 2026, 12:01 AM UTC

  1. investigating Apr 27, 2026, 04:38 PM UTC

    GitHub is reporting a GitHub Actions degradation, and we're monitoring the impact

  2. resolved Apr 28, 2026, 12:01 AM UTC

    GitHub resolved their upstream outage and we are not seeing any customer impact.

Read the full incident report →

Minor April 23, 2026 Github Actions and Packages degraded 2h 34m Overall, EU Central (eu-central ARM), EU Central (eu-central x86), US West (us-west ARM) and 3 more

Started Apr 23, 2026, 02:57 PM UTC· resolved Apr 23, 2026, 05:31 PM UTC

  1. investigating Apr 23, 2026, 02:57 PM UTC

    We are currently monitoring the effects of a Github outage:

  2. resolved Apr 23, 2026, 03:24 PM UTC

    This incident has been resolved by Github.

  3. monitoring Apr 23, 2026, 03:28 PM UTC

    It seems that there's still trailing effects from the incident and we're seeing queue times due to slow webhook delivery.

  4. resolved Apr 23, 2026, 05:31 PM UTC

    This incident has been resolved by Github.

Read the full incident report →

Minor April 22, 2026 Sticky Disk degradation in the US-WEST cluster 2h 23m Overall, Incremental Docker Builders, Docker Container Cache, Sticky Disks

Started Apr 22, 2026, 05:50 PM UTC· resolved Apr 22, 2026, 08:14 PM UTC

  1. investigating Apr 22, 2026, 05:50 PM UTC

    We are currently investigating this incident.

  2. monitoring Apr 22, 2026, 06:41 PM UTC

    We needed to perform an emergency maintenance operation in our storage cluster. We are observing the recovery and seeing docker build speeds improve.

  3. resolved Apr 22, 2026, 08:14 PM UTC

    This incident has been resolved.

Read the full incident report →

Major April 20, 2026 Control Plane Outage Job Adoption affected – Overall

Started Apr 20, 2026, 04:35 PM UTC· resolved Apr 20, 2026, 04:35 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Major April 20, 2026 Control Plane Outage affecting Job Adoption – Overall

Started Apr 20, 2026, 03:12 PM UTC· resolved Apr 20, 2026, 03:12 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Minor April 16, 2026 Job queue backlog being processed – Overall

Started Apr 16, 2026, 05:00 PM UTC· resolved Apr 16, 2026, 05:00 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Minor April 13, 2026 Elevated Queue Times for US x86 Runners – Overall

Started Apr 13, 2026, 06:41 PM UTC· resolved Apr 13, 2026, 06:41 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Minor April 12, 2026 Storage cluster expansion in US-West – Overall

Started Apr 12, 2026, 04:00 AM UTC· resolved Apr 12, 2026, 04:00 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Minor March 24, 2026 GitHub Actions & Webhooks Degraded Performance – Overall

Started Mar 24, 2026, 08:28 PM UTC· resolved Mar 24, 2026, 08:28 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →