Is Blacksmith down?

Jobs on Linux runners in us-east are seeing severely slow downloads from GitHub and Google services and in us-west slow downloads from GitHub. This includes actions/checkout, git clone/git fetch, downloading actions from github.com, and pulls from Google Container Registry / Artifact Registry. Jobs that depend on these may run much longer than usual or time out. The job slowdown is also leading to increased job adoption times in those regions.

Download speeds for jobs in us-east have recovered and the region is back to operational. Jobs in us-west are still seeing slow downloads from GitHub while our network provider works on the issue.

Download speeds in us-west have recovered and both US regions are back to operational. Affected jobs can be re-run and should complete normally. We are continuing to monitor and will post an update once we have confirmed stability.

This incident has been resolved. Download speeds in us-east and us-west are back to normal, and affected jobs can be re-run.

An issue with our upstream macOS provider has temporarily reduced our available macOS capacity, and we are working with them to bring machines back online. Customers may see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected; Linux and Windows runners are unaffected. We will provide an update shortly.

We are continuing to bring additional macOS hosts online with our upstream provider however available macOS capacity has not yet recovered. Customers can still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected. We will provide another update shortly

We are continuing to work with our upstream provider to bring the affected macOS hosts back online. Customers should still expect longer than normal queue times for macOS jobs. We will provide another update shortly.

We have restored a portion of the affected macOS capacity and are continuing to work with our upstream provider to bring the remaining hosts back online. Customers should still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

We are continuing to bring additional macOS hosts back online with our upstream provider, and the capacity restored earlier remains in service. Customers may still see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

We have returned an additional group of macOS hosts to service and expect a further large batch to come online within the next 30 minutes, at which point waiting macOS jobs should start being picked up more quickly. Customers should currently expect 6vcpu macOS jobs to wait around 5 minutes to start, while 12vcpu macOS jobs are waiting considerably longer, and Linux and Windows runners are unaffected.

We are continuing to bring macOS hosts back into service in batches and are working with our upstream provider to release the remaining hosts, but macOS demand still exceeds the capacity online. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners remain unaffected.

We are continuing to bring macOS hosts back into service and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer. Linux and Windows runners are unaffected.

We have brought additional macOS capacity online and queue times have started to improve. Customers should currently expect 6vcpu macOS jobs to wait a few minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners are unaffected.

6vcpu macOS jobs are now starting within a few minutes, and the backlog of 12vcpu macOS jobs is draining as we continue to add capacity with our upstream provider. Customers with 12vcpu macOS jobs already queued may still see longer waits for them to start, while new jobs are picking up faster, and Linux and Windows runners are unaffected

New macOS jobs are starting within a few minutes for most customers, though some 6vcpu macOS jobs may wait up to 10 minutes during busy periods while we finish restoring capacity with our upstream provider. A backlog of 12vcpu macOS jobs queued earlier today is still being worked through, and those jobs may take longer to start, while Linux and Windows runners are unaffected.

The queue for macOS jobs is subsiding as restored capacity continues to come online with our upstream provider, and new 6vcpu and 12vcpu macOS jobs are picking up normally for most customers. A small number of 12vcpu macOS jobs queued earlier today are still being worked through and may take longer to start, while Linux and Windows runners are unaffected. We are continuing to monitor and will resolve this incident once the remaining backlog has cleared.

The macOS job queue has cleared and new 6vcpu and 12vcpu macOS jobs are picking up normally across all regions. We are continuing to restore the remaining capacity with our upstream provider and will keep monitoring before resolving this incident. Linux and Windows runners were unaffected throughout.

This incident is resolved. macOS capacity has been restored and macOS jobs have been picking up normally since this evening.

An issue with our upstream macOS provider has temporarily reduced our available macOS capacity, and we are working with them to bring machines back online. Customers may see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected; Linux and Windows runners are unaffected. We will provide an update shortly..

We are continuing to bring additional macOS hosts online with our upstream provider however available macOS capacity has not yet recovered. Customers can still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected. We will provide another update shortly.

We are continuing to work with our upstream provider to bring the affected macOS hosts back online. Customers should still expect longer than normal queue times for macOS jobs. We will provide another update shortly..

We have restored a portion of the affected macOS capacity and are continuing to work with our upstream provider to bring the remaining hosts back online. Customers should still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected..

We are continuing to bring additional macOS hosts back online with our upstream provider, and the capacity restored earlier remains in service. Customers may still see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected..

We have returned an additional group of macOS hosts to service and expect a further large batch to come online within the next 30 minutes, at which point waiting macOS jobs should start being picked up more quickly. Customers should currently expect 6vcpu macOS jobs to wait around 5 minutes to start, while 12vcpu macOS jobs are waiting considerably longer, and Linux and Windows runners are unaffected..

We are continuing to bring macOS hosts back into service in batches and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners remain unaffected..

We are continuing to bring macOS hosts back into service and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer. Linux and Windows runners are unaffected..

We have brought additional macOS capacity online and queue times have started to improve. Customers should currently expect 6vcpu macOS jobs to wait a few minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners are unaffected. .

6vcpu macOS jobs are now starting within a few minutes, and the backlog of 12vcpu macOS jobs is draining as we continue to add capacity with our upstream provider. Customers with 12vcpu macOS jobs already queued may still see longer waits for them to start, while new jobs are picking up faster, and Linux and Windows runners are unaffected.

New macOS jobs are starting within a few minutes for most customers, though some 6vcpu macOS jobs may wait up to 10 minutes during busy periods while we finish restoring capacity with our upstream provider. A backlog of 12vcpu macOS jobs queued earlier today is still being worked through, and those jobs may take longer to start, while Linux and Windows runners are unaffected..

The queue for macOS jobs is subsiding as restored capacity continues to come online with our upstream provider, and new 6vcpu and 12vcpu macOS jobs are picking up normally for most customers. A small number of 12vcpu macOS jobs queued earlier today are still being worked through and may take longer to start, while Linux and Windows runners are unaffected. We are continuing to monitor and will resolve this incident once the remaining backlog has cleared..

The macOS job queue has cleared and new 6vcpu and 12vcpu macOS jobs are picking up normally across all regions. We are continuing to restore the remaining capacity with our upstream provider and will keep monitoring before resolving this incident. Linux and Windows runners were unaffected throughout..

This incident is resolved. macOS capacity has been restored and macOS jobs have been picking up normally since this evening..

We're currently investigating an issue with our dashboard. Certain observability views may take long to load or fail to load completely.

We've applied a few mitigations and are seeing some improvement. Certain queries are still timing out. We're still investigating.

We've rolled out a fix and are currently monitoring the results. We are seeing major recovery in the request latencies in our dashboard.

This incident has been resolved.

We're currently investigating an issue with our dashboard. Certain observability views may take long to load or fail to load completely. .

We've applied a few mitigations and are seeing some improvement. Certain queries are still timing out. We're still investigating..

We've rolled out a fix and are currently monitoring the results. We are seeing major recovery in the request latencies in our dashboard..

This incident has been resolved..