Blacksmith Outage History

Started Oct 27, 2026, 07:13 PM UTC· resolved Oct 27, 2026, 08:20 PM UTC

1. investigating Oct 27, 2026, 07:13 PM UTC

   Jobs on Linux runners in us-east are seeing severely slow downloads from GitHub and Google services and in us-west slow downloads from GitHub. This includes actions/checkout, git clone/git fetch, downloading actions from github.com, and pulls from Google Container Registry / Artifact Registry. Jobs that depend on these may run much longer than usual or time out. The job slowdown is also leading to increased job adoption times in those regions.

2. identified Oct 27, 2026, 07:31 PM UTC

   Download speeds for jobs in us-east have recovered and the region is back to operational. Jobs in us-west are still seeing slow downloads from GitHub while our network provider works on the issue.

3. monitoring Oct 27, 2026, 07:39 PM UTC

   Download speeds in us-west have recovered and both US regions are back to operational. Affected jobs can be re-run and should complete normally. We are continuing to monitor and will post an update once we have confirmed stability.

4. resolved Oct 27, 2026, 08:20 PM UTC

   This incident has been resolved. Download speeds in us-east and us-west are back to normal, and affected jobs can be re-run.

Read the full incident report →

Started Oct 20, 2026, 10:18 AM UTC· resolved Oct 21, 2026, 12:23 AM UTC

1. investigating Oct 20, 2026, 10:18 AM UTC

   An issue with our upstream macOS provider has temporarily reduced our available macOS capacity, and we are working with them to bring machines back online. Customers may see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected; Linux and Windows runners are unaffected. We will provide an update shortly..

2. investigating Oct 20, 2026, 10:18 AM UTC

   An issue with our upstream macOS provider has temporarily reduced our available macOS capacity, and we are working with them to bring machines back online. Customers may see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected; Linux and Windows runners are unaffected. We will provide an update shortly.

3. investigating Oct 20, 2026, 11:02 AM UTC

   We are continuing to bring additional macOS hosts online with our upstream provider however available macOS capacity has not yet recovered. Customers can still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected. We will provide another update shortly.

4. investigating Oct 20, 2026, 11:02 AM UTC

   We are continuing to bring additional macOS hosts online with our upstream provider however available macOS capacity has not yet recovered. Customers can still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected. We will provide another update shortly

5. investigating Oct 20, 2026, 12:37 PM UTC

   We are continuing to work with our upstream provider to bring the affected macOS hosts back online. Customers should still expect longer than normal queue times for macOS jobs. We will provide another update shortly..

6. investigating Oct 20, 2026, 12:37 PM UTC

   We are continuing to work with our upstream provider to bring the affected macOS hosts back online. Customers should still expect longer than normal queue times for macOS jobs. We will provide another update shortly.

7. investigating Oct 20, 2026, 01:19 PM UTC

   We have restored a portion of the affected macOS capacity and are continuing to work with our upstream provider to bring the remaining hosts back online. Customers should still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected..

8. investigating Oct 20, 2026, 01:19 PM UTC

   We have restored a portion of the affected macOS capacity and are continuing to work with our upstream provider to bring the remaining hosts back online. Customers should still expect longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

9. investigating Oct 20, 2026, 01:57 PM UTC

   We are continuing to bring additional macOS hosts back online with our upstream provider, and the capacity restored earlier remains in service. Customers may still see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected..

10. investigating Oct 20, 2026, 01:57 PM UTC

    We are continuing to bring additional macOS hosts back online with our upstream provider, and the capacity restored earlier remains in service. Customers may still see longer than normal queue times for macOS jobs, with 12vcpu macOS jobs most affected, while Linux and Windows runners are unaffected.

11. investigating Oct 20, 2026, 02:59 PM UTC

    We have returned an additional group of macOS hosts to service and expect a further large batch to come online within the next 30 minutes, at which point waiting macOS jobs should start being picked up more quickly. Customers should currently expect 6vcpu macOS jobs to wait around 5 minutes to start, while 12vcpu macOS jobs are waiting considerably longer, and Linux and Windows runners are unaffected..

12. investigating Oct 20, 2026, 02:59 PM UTC

    We have returned an additional group of macOS hosts to service and expect a further large batch to come online within the next 30 minutes, at which point waiting macOS jobs should start being picked up more quickly. Customers should currently expect 6vcpu macOS jobs to wait around 5 minutes to start, while 12vcpu macOS jobs are waiting considerably longer, and Linux and Windows runners are unaffected.

13. investigating Oct 20, 2026, 04:05 PM UTC

    We are continuing to bring macOS hosts back into service in batches and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners remain unaffected..

14. investigating Oct 20, 2026, 04:05 PM UTC

    We are continuing to bring macOS hosts back into service in batches and are working with our upstream provider to release the remaining hosts, but macOS demand still exceeds the capacity online. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners remain unaffected.

15. identified Oct 20, 2026, 05:04 PM UTC

    We are continuing to bring macOS hosts back into service and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer. Linux and Windows runners are unaffected..

16. identified Oct 20, 2026, 05:04 PM UTC

    We are continuing to bring macOS hosts back into service and are working with our upstream provider to release the remaining hosts. Customers should currently expect 6vcpu macOS jobs to wait around 10 minutes to start and 12vcpu macOS jobs considerably longer. Linux and Windows runners are unaffected.

17. identified Oct 20, 2026, 06:29 PM UTC

    We have brought additional macOS capacity online and queue times have started to improve. Customers should currently expect 6vcpu macOS jobs to wait a few minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners are unaffected. .

18. identified Oct 20, 2026, 06:29 PM UTC

    We have brought additional macOS capacity online and queue times have started to improve. Customers should currently expect 6vcpu macOS jobs to wait a few minutes to start and 12vcpu macOS jobs considerably longer, while Linux and Windows runners are unaffected.

19. identified Oct 20, 2026, 07:30 PM UTC

    6vcpu macOS jobs are now starting within a few minutes, and the backlog of 12vcpu macOS jobs is draining as we continue to add capacity with our upstream provider. Customers with 12vcpu macOS jobs already queued may still see longer waits for them to start, while new jobs are picking up faster, and Linux and Windows runners are unaffected.

20. identified Oct 20, 2026, 07:30 PM UTC

    6vcpu macOS jobs are now starting within a few minutes, and the backlog of 12vcpu macOS jobs is draining as we continue to add capacity with our upstream provider. Customers with 12vcpu macOS jobs already queued may still see longer waits for them to start, while new jobs are picking up faster, and Linux and Windows runners are unaffected

21. monitoring Oct 20, 2026, 08:43 PM UTC

    New macOS jobs are starting within a few minutes for most customers, though some 6vcpu macOS jobs may wait up to 10 minutes during busy periods while we finish restoring capacity with our upstream provider. A backlog of 12vcpu macOS jobs queued earlier today is still being worked through, and those jobs may take longer to start, while Linux and Windows runners are unaffected..

22. monitoring Oct 20, 2026, 08:43 PM UTC

    New macOS jobs are starting within a few minutes for most customers, though some 6vcpu macOS jobs may wait up to 10 minutes during busy periods while we finish restoring capacity with our upstream provider. A backlog of 12vcpu macOS jobs queued earlier today is still being worked through, and those jobs may take longer to start, while Linux and Windows runners are unaffected.

23. monitoring Oct 20, 2026, 10:02 PM UTC

    The queue for macOS jobs is subsiding as restored capacity continues to come online with our upstream provider, and new 6vcpu and 12vcpu macOS jobs are picking up normally for most customers. A small number of 12vcpu macOS jobs queued earlier today are still being worked through and may take longer to start, while Linux and Windows runners are unaffected. We are continuing to monitor and will resolve this incident once the remaining backlog has cleared..

24. monitoring Oct 20, 2026, 10:02 PM UTC

    The queue for macOS jobs is subsiding as restored capacity continues to come online with our upstream provider, and new 6vcpu and 12vcpu macOS jobs are picking up normally for most customers. A small number of 12vcpu macOS jobs queued earlier today are still being worked through and may take longer to start, while Linux and Windows runners are unaffected. We are continuing to monitor and will resolve this incident once the remaining backlog has cleared.

25. monitoring Oct 20, 2026, 11:16 PM UTC

    The macOS job queue has cleared and new 6vcpu and 12vcpu macOS jobs are picking up normally across all regions. We are continuing to restore the remaining capacity with our upstream provider and will keep monitoring before resolving this incident. Linux and Windows runners were unaffected throughout..

26. monitoring Oct 20, 2026, 11:16 PM UTC

    The macOS job queue has cleared and new 6vcpu and 12vcpu macOS jobs are picking up normally across all regions. We are continuing to restore the remaining capacity with our upstream provider and will keep monitoring before resolving this incident. Linux and Windows runners were unaffected throughout.

27. resolved Oct 21, 2026, 12:23 AM UTC

    This incident is resolved. macOS capacity has been restored and macOS jobs have been picking up normally since this evening..

28. resolved Oct 21, 2026, 12:23 AM UTC

    This incident is resolved. macOS capacity has been restored and macOS jobs have been picking up normally since this evening.

Read the full incident report →

Started Oct 13, 2026, 10:01 PM UTC· resolved Oct 13, 2026, 11:04 PM UTC

1. investigating Oct 13, 2026, 10:01 PM UTC

   We're currently investigating an issue with our dashboard. Certain observability views may take long to load or fail to load completely. .

2. investigating Oct 13, 2026, 10:01 PM UTC

   We're currently investigating an issue with our dashboard. Certain observability views may take long to load or fail to load completely.

3. investigating Oct 13, 2026, 10:30 PM UTC

   We've applied a few mitigations and are seeing some improvement. Certain queries are still timing out. We're still investigating..

4. investigating Oct 13, 2026, 10:30 PM UTC

   We've applied a few mitigations and are seeing some improvement. Certain queries are still timing out. We're still investigating.

5. monitoring Oct 13, 2026, 10:42 PM UTC

   We've rolled out a fix and are currently monitoring the results. We are seeing major recovery in the request latencies in our dashboard..

6. monitoring Oct 13, 2026, 10:42 PM UTC

   We've rolled out a fix and are currently monitoring the results. We are seeing major recovery in the request latencies in our dashboard.

7. resolved Oct 13, 2026, 11:04 PM UTC

   This incident has been resolved..

8. resolved Oct 13, 2026, 11:04 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 29, 2026, 09:10 PM UTC· resolved Sep 29, 2026, 11:04 PM UTC

1. investigating Sep 29, 2026, 09:10 PM UTC

   Jobs across all regions are taking longer to start and cache operations are failing. We are continuing to investigate the root cause..

2. investigating Sep 29, 2026, 09:10 PM UTC

   Jobs across all regions are taking longer to start and cache operations are failing. We are continuing to investigate the root cause.

3. identified Sep 29, 2026, 09:28 PM UTC

   We have applied a fix and are monitoring its effect. Customers may still see delayed job starts and failing cache operations while recovery completes. We will provide an update within the next 30 minutes..

4. identified Sep 29, 2026, 09:28 PM UTC

   We have applied a fix and are monitoring its effect. Customers may still see delayed job starts and failing cache operations while recovery completes. We will provide an update within the next 30 minutes.

5. monitoring Sep 29, 2026, 09:43 PM UTC

   Our mitigation has taken effect and job start times and cache operations are improving. Customers may still see some delayed job starts and cache failures while recovery completes. We will provide an update within the next 30 minutes..

6. monitoring Sep 29, 2026, 09:43 PM UTC

   Our mitigation has taken effect and job start times and cache operations are improving. Customers may still see some delayed job starts and cache failures while recovery completes. We will provide an update within the next 30 minutes.

7. monitoring Sep 29, 2026, 10:20 PM UTC

   Caching has recovered, and job queue times across EU regions have recovered. Queue times for larger jobs (16 and 32 vcpu jobs) in US west and US east continue to remain elevated. We are continuing to monitor for any regressions..

8. monitoring Sep 29, 2026, 10:20 PM UTC

   Caching has recovered, and job queue times across EU regions have recovered. Queue times for larger jobs (16 and 32 vcpu jobs) in US west and US east continue to remain elevated. We are continuing to monitor for any regressions.

9. monitoring Sep 29, 2026, 10:42 PM UTC

   This incident is resolved. Job start times and cache operations returned to normal in all regions, and we are continuing to monitor. Jobs that failed during the incident can be re-run..

10. monitoring Sep 29, 2026, 10:42 PM UTC

    This incident is resolved. Job starts and cache operations returned to normal in all regions, and we are continuing to monitor. Jobs that failed during the incident can be re-run.

11. resolved Sep 29, 2026, 11:04 PM UTC

    This incident has been resolved..

12. resolved Sep 29, 2026, 11:04 PM UTC

    This incident has been resolved.

Read the full incident report →

Started Sep 29, 2026, 05:00 PM UTC· resolved Sep 29, 2026, 06:32 PM UTC

1. investigating Sep 29, 2026, 05:00 PM UTC

   We are investigating failing GitHub Actions cache operations for jobs running in us-west since approximately 16:10 UTC. Affected jobs may see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected. .

2. investigating Sep 29, 2026, 05:00 PM UTC

   We are investigating failing GitHub Actions cache operations for jobs running in us-west since approximately 16:10 UTC. Affected jobs may see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected.

3. identified Sep 29, 2026, 05:09 PM UTC

   We have identified the cause of the failures, and we are implementing a fix to restore cache service. Customers with jobs in us-west may still see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected..

4. identified Sep 29, 2026, 05:09 PM UTC

   We have identified the cause of the failures, and we are implementing a fix to restore cache service. Customers with jobs in us-west may still see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected.

5. monitoring Sep 29, 2026, 05:35 PM UTC

   We have restored the cache infrastructure in us-west, and cache failures have decreased but are not yet back to normal. Jobs in us-west may still see some cache restores and saves fail and fall back to a full install, while the earlier delayed job starts have cleared and other regions are unaffected. We are bringing additional cache capacity online to fully restore service..

6. monitoring Sep 29, 2026, 05:35 PM UTC

   We have restored the cache infrastructure in us-west, and cache failures have decreased but are not yet back to normal. Jobs in us-west may still see some cache restores and saves fail and fall back to a full install, while the earlier delayed job starts have cleared and other regions are unaffected. We are bringing additional cache capacity online to fully restore service.

7. monitoring Sep 29, 2026, 06:06 PM UTC

   Cache errors for jobs in us-west have largely subsided. As we complete recovery, some jobs may see a one-time cache miss on their next run. We are monitoring and will provide an update within the next hour..

8. monitoring Sep 29, 2026, 06:06 PM UTC

   Cache errors for jobs in us-west have largely subsided. As we complete recovery, some jobs may see a one-time cache miss on their next run. We are monitoring and will provide an update within the next hour.

9. resolved Sep 29, 2026, 06:32 PM UTC

   This incident is resolved. GitHub Actions cache operations for jobs in us-west have returned to normal, and any jobs that failed during the incident can be re-run..

10. resolved Sep 29, 2026, 06:32 PM UTC

    This incident is resolved. GitHub Actions cache operations for jobs in us-west have returned to normal, and any jobs that failed during the incident can be re-run.

Read the full incident report →

Started Sep 28, 2026, 08:52 PM UTC· resolved Sep 28, 2026, 10:14 PM UTC

1. identified Sep 28, 2026, 08:52 PM UTC

   A rack-level power failure at our us-west data center caused some jobs to fail with runner communication errors. The issue has been communicated with our upstream provider and we are identifying the affected machines to mitigate the impact to our customers.

2. monitoring Sep 28, 2026, 09:53 PM UTC

   The affected rack remains offline while our upstream provider works to restore its power, and jobs in us-west are running normally on the rest of the fleet. Any jobs that failed with runner communication errors can be safely re-run.

3. resolved Sep 28, 2026, 10:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 24, 2026, 07:16 PM UTC· resolved Sep 24, 2026, 07:50 PM UTC

1. investigating Sep 24, 2026, 07:16 PM UTC

   We are investigating degraded network connectivity between our us-west region and GitHub. Customers running jobs in us-west may see slower-than-normal git checkouts, while jobs in other regions are not affected. Workflows using the Blacksmith checkout action with git checkout caching are less affected, and setup instructions are in the documentation below. Git checkout caching documentation: .

2. investigating Sep 24, 2026, 07:16 PM UTC

   We are investigating degraded network connectivity between our us-west region and GitHub. Customers running jobs in us-west may see slower-than-normal git checkouts, while jobs in other regions are not affected.

3. monitoring Sep 24, 2026, 07:31 PM UTC

   After rerouting traffic, congestion on the upstream network link has subsided, and git checkout performance in us-west has returned to normal. We are monitoring and will resolve this incident once performance has continued to stay stable. .

4. monitoring Sep 24, 2026, 07:31 PM UTC

   After rerouting traffic, congestion on the upstream network link has subsided, and git checkout performance in us-west has returned to normal. We are monitoring and will resolve this incident once performance has continued to stay stable.

5. resolved Sep 24, 2026, 07:50 PM UTC

   Git checkout performance in us-west has remained stable since we rerouted traffic away from the congested upstream network link, and this incident is now resolved..

6. resolved Sep 24, 2026, 07:50 PM UTC

   Git checkout performance in us-west has remained stable since we rerouted traffic away from the congested upstream network link, and this incident is now resolved.

Read the full incident report →

Started Sep 21, 2026, 06:37 PM UTC· resolved Sep 21, 2026, 06:47 PM UTC

1. identified Sep 21, 2026, 06:37 PM UTC

   We are seeing elevated errors in job adoption and other control plane APIs. This is yielding higher latencies for your job to be adopted. You may also see certain actions fail, such as actions caching and sticky disks. We have identified the issue and are rolling out a mitigation..

2. identified Sep 21, 2026, 06:37 PM UTC

   We are seeing elevated errors in job adoption and other control plane APIs. This is yielding higher latencies for your job to be adopted. You may also see certain actions fail, such as actions caching and sticky disks. We have identified the issue and are rolling out a mitigation.

3. resolved Sep 21, 2026, 06:47 PM UTC

   This issue has been resolved, and all services have operating normally. If any of your jobs failed during this time, please re-run them..

4. resolved Sep 21, 2026, 06:47 PM UTC

   This issue has been resolved, and all services have operating normally. If any of your jobs failed during this time, please re-run them.

Read the full incident report →

Started Sep 13, 2026, 01:11 AM UTC· resolved Sep 13, 2026, 01:58 AM UTC

1. investigating Sep 13, 2026, 01:11 AM UTC

   We are currently observing network stalling in our US East region with runner connectivity against AWS's us-east-1 region. Uploads and downloads to and from ECR and S3 may experience occasional stalling. We are investigating..

2. investigating Sep 13, 2026, 01:11 AM UTC

   We are currently observing network stalling in our US East region with runner connectivity against AWS's us-east-1 region. Uploads and downloads to and from ECR and S3 may experience occasional stalling. We are investigating.

3. monitoring Sep 13, 2026, 01:40 AM UTC

   We have identified a small subset of runners that are exhibiting this network stalling and have isolated them. We're continuing to monitor recovery..

4. monitoring Sep 13, 2026, 01:40 AM UTC

   We have identified a small subset of runners that are exhibiting this network stalling and have isolated them. We're continuing to monitor recovery.

5. resolved Sep 13, 2026, 01:58 AM UTC

   This incident has been resolved..

6. resolved Sep 13, 2026, 01:58 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 11, 2026, 05:30 AM UTC· resolved Sep 11, 2026, 02:59 PM UTC

1. monitoring Sep 11, 2026, 05:30 AM UTC

   Since 06:30 UTC Canonical has had three outages affecting [archive.ubuntu.com](http://archive.ubuntu.com), [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.

2. monitoring Sep 11, 2026, 05:30 AM UTC

   Since 06:30 UTC Canonical has had three outages affecting [archive.ubuntu.com](http://archive.ubuntu.com), [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs..

3. monitoring Sep 11, 2026, 02:21 PM UTC

   We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs..

4. monitoring Sep 11, 2026, 02:21 PM UTC

   We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.

5. resolved Sep 11, 2026, 02:59 PM UTC

   This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future..

6. resolved Sep 11, 2026, 02:59 PM UTC

   This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future.

Read the full incident report →

Started Sep 09, 2026, 04:46 PM UTC· resolved Sep 09, 2026, 06:06 PM UTC

1. investigating Sep 09, 2026, 04:46 PM UTC

   We are currently experiencing a major outage with both our caching infrastructure and our dashboards. These components are affected across all regions. We are investigating the issue and will provide an update soon..

2. investigating Sep 09, 2026, 05:14 PM UTC

   We are currently experiencing a major outage with both our caching infrastructure and our dashboards. These components are affected across all regions. We are investigating the issue and will provide an update soon.

3. monitoring Sep 09, 2026, 05:21 PM UTC

   We are seeing recovery of our caching infrastructure and our dashboard. We will continue to monitor this situation closely..

4. monitoring Sep 09, 2026, 05:21 PM UTC

   We are seeing recovery of our caching infrastructure and our dashboard. We will continue to monitor this situation closely.

5. resolved Sep 09, 2026, 06:06 PM UTC

   This incident has been resolved..

6. resolved Sep 09, 2026, 06:06 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 08, 2026, 03:00 PM UTC· resolved Sep 08, 2026, 04:44 PM UTC

1. investigating Sep 08, 2026, 03:00 PM UTC

   Since approximately 12:20 UTC, some jobs in our eu-west region have been failing with the GitHub error "The self-hosted runner lost communication with the server"; other regions are unaffected. Affected jobs can be safely re-run while we continue to investigate the underlying cause. .

2. investigating Sep 08, 2026, 03:00 PM UTC

   Since approximately 12:20 UTC, some jobs in our eu-west region have been failing with the GitHub error "The self-hosted runner lost communication with the server"; other regions are unaffected. Affected jobs can be safely re-run while we continue to investigate the underlying cause.

3. monitoring Sep 08, 2026, 03:31 PM UTC

   Between approximately 12:20 and 15:15 UTC, some jobs in our eu-west region failed with the GitHub error "The self-hosted runner lost communication with the server"; other regions were unaffected. Job failures in eu-west returned to normal levels as of 15:15 UTC, and any affected jobs can be safely re-run while we continue to investigate the underlying cause..

4. monitoring Sep 08, 2026, 03:31 PM UTC

   Between approximately 12:20 and 15:15 UTC, some jobs in our eu-west region failed with the GitHub error "The self-hosted runner lost communication with the server"; other regions were unaffected. Job failures in eu-west returned to normal levels as of 15:15 UTC, and any affected jobs can be safely re-run while we continue to investigate the underlying cause.

5. resolved Sep 08, 2026, 04:44 PM UTC

   This incident is resolved. Job failures in our eu-west region returned to normal levels at 15:15 UTC and have remained there since; any jobs that failed between approximately 12:20 and 15:15 UTC with the error "The self-hosted runner lost communication with the server" can be safely re-run..

6. resolved Sep 08, 2026, 04:44 PM UTC

   This incident is resolved. Job failures in our eu-west region returned to normal levels at 15:15 UTC and have remained there since; any jobs that failed between approximately 12:20 and 15:15 UTC with the error "The self-hosted runner lost communication with the server" can be safely re-run.

Read the full incident report →

Started Sep 02, 2026, 04:13 PM UTC· resolved Sep 02, 2026, 07:23 PM UTC

1. investigating Sep 02, 2026, 04:13 PM UTC

   We are investigating reports of higher latency on sticky disk operations in our us-west region. Customers running jobs in us-west may see slower incremental Docker builds, Git caching, and container caching, so affected jobs can take longer than usual to complete. .

2. investigating Sep 02, 2026, 04:13 PM UTC

   We are investigating reports of higher latency on sticky disk operations in our us-west region. Customers running jobs in us-west may see slower incremental Docker builds, Git caching, and container caching, so affected jobs can take longer than usual to complete.

3. investigating Sep 02, 2026, 04:48 PM UTC

   Sticky disk storage in our us-west region is experiencing higher latency; our other regions are not affected. Customers running jobs in us-west may see slower Git-cached checkouts, incremental Docker builds, and container caching. We are continuing to investigate the underlying cause and will provide another update within the next 30 minutes..

4. investigating Sep 02, 2026, 04:48 PM UTC

   Sticky disk storage in our us-west region is experiencing higher latency; our other regions are not affected. Customers running jobs in us-west may see slower Git-cached checkouts, incremental Docker builds, and container caching. We are continuing to investigate the underlying cause and will provide another update within the next 30 minutes.

5. investigating Sep 02, 2026, 05:22 PM UTC

   We are still working to resolve elevated latency on sticky disk storage in our us-west region; other regions are not affected. Customers running jobs in us-west may continue to see slower Git-cached checkouts, incremental Docker builds, and container caching. Our investigation into the underlying cause is ongoing and we will provide another update within the next 30 minutes. .

6. investigating Sep 02, 2026, 05:22 PM UTC

   We are still working to resolve elevated latency on sticky disk storage in our us-west region; other regions are not affected. Customers running jobs in us-west may continue to see slower Git-cached checkouts, incremental Docker builds, and container caching. Our investigation into the underlying cause is ongoing and we will provide another update within the next 30 minutes.

7. monitoring Sep 02, 2026, 06:10 PM UTC

   Our us-west region storage cluster's latency and error rates have returned to baseline as of approximately 17:15 UTC, following mitigations that reduce load on the affected storage. Git-cached checkouts, incremental Docker builds, and container caching in us-west are back to normal, and we are monitoring to confirm the recovery holds while we continue to investigate the underlying cause. We will provide a final update within the next hour..

8. monitoring Sep 02, 2026, 06:10 PM UTC

   Our us-west region storage cluster's latency and error rates have returned to baseline as of approximately 17:15 UTC, following mitigations that reduce load on the affected storage. Git-cached checkouts, incremental Docker builds, and container caching in us-west are back to normal, and we are monitoring to confirm the recovery holds while we continue to investigate the underlying cause. We will provide a final update within the next hour.

9. resolved Sep 02, 2026, 07:23 PM UTC

   This incident is resolved. Sticky disk storage in our us-west region came under more read load than it could serve at normal latency, which slowed and produced higher error rates for Git-cached checkouts, incremental Docker builds, and container caching. We reduced and redistributed that load, and performance has been normal since approximately 17:15 UTC. Jobs that failed during the incident can be safely re-run. .

10. resolved Sep 02, 2026, 07:23 PM UTC

    This incident is resolved. Sticky disk storage in our us-west region came under more read load than it could serve at normal latency, which slowed and produced higher error rates for Git-cached checkouts, incremental Docker builds, and container caching. We reduced and redistributed that load, and performance has been normal since approximately 17:15 UTC. Jobs that failed during the incident can be safely re-run.

Read the full incident report →

Started Sep 02, 2026, 02:09 PM UTC· resolved Sep 02, 2026, 04:30 PM UTC

1. investigating Sep 02, 2026, 02:09 PM UTC

   We are currently observing higher rates of errors for the upstream GHCR registries which may indicate an undeclared GitHub incident. We are currently monitoring..

2. investigating Sep 02, 2026, 02:09 PM UTC

   We are currently observing higher rates of errors for the upstream GHCR registries which may indicate an undeclared GitHub incident. We are currently monitoring.

3. investigating Sep 02, 2026, 02:20 PM UTC

   We are seeing evidence that Git Checkouts are also affected by this. We are seeing high TCP retransmit rates into the EU GitHub loadbalancer. Checkouts in the EU West, EU Central, US East regions are affected.

4. investigating Sep 02, 2026, 02:20 PM UTC

   We are seeing evidence that Git Checkouts are also affected by this. We are seeing high TCP retransmit rates into the EU GitHub loadbalancer. Checkouts in the EU West, EU Central, US East regions are affected..

5. monitoring Sep 02, 2026, 03:43 PM UTC

   Connections to [github.com](http://github.com) and [ghcr.io](http://ghcr.io) from our eu-west, eu-central, and us-east regions have returned to normal as of approximately 15:00 UTC, so checkouts are completing at normal speed and login failures to GitHub Container Registry have dropped to baseline levels. We are continuing to monitor, and re-running any jobs that failed during this period should succeed..

6. monitoring Sep 02, 2026, 03:43 PM UTC

   Connections to [github.com](http://github.com) and [ghcr.io](http://ghcr.io) from our eu-west, eu-central, and us-east regions have returned to normal as of approximately 15:00 UTC, so checkouts are completing at normal speed and login failures to GitHub Container Registry have dropped to baseline levels. We are continuing to monitor, and re-running any jobs that failed during this period should succeed.

7. resolved Sep 02, 2026, 04:30 PM UTC

   This incident has been resolved. GitHub connectivity from our EU regions was degraded from \~12:45 to 16:15 UTC, affecting checkouts, [ghcr.io](http://ghcr.io), and some eu-west job starts. Normal since 16:15\. Affected jobs can be re-run..

8. resolved Sep 02, 2026, 04:30 PM UTC

   This incident has been resolved. GitHub connectivity from our EU regions was degraded from \~12:45 to 16:15 UTC, affecting checkouts, [ghcr.io](http://ghcr.io), and some eu-west job starts. Normal since 16:15\. Affected jobs can be re-run.

Read the full incident report →

Started Aug 18, 2026, 08:45 PM UTC· resolved Aug 18, 2026, 11:14 PM UTC

1. monitoring Aug 18, 2026, 08:45 PM UTC

   Jobs failure rate at an elevated rate caused by an upstream GitHub being rejected by 429 rate-limit errors. We are seeing a single-digit percentage increase in job failure rate across all organizations and are seeing similar failures on non-Blacksmith infrastructure..

2. monitoring Aug 18, 2026, 08:45 PM UTC

   Jobs failure rate at an elevated rate caused by an upstream GitHub being rejected by 429 rate-limit errors. We are seeing a single-digit percentage increase in job failure rate across all organizations and are seeing similar failures on non-Blacksmith infrastructure.

3. resolved Aug 18, 2026, 11:14 PM UTC

   This incident has been resolved..

4. resolved Aug 18, 2026, 11:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 17, 2026, 06:58 PM UTC· resolved Aug 17, 2026, 07:00 PM UTC

1. investigating Aug 17, 2026, 06:58 PM UTC

   We are seeing an increased error rate processing webhook events which will lead to delays in adoption jobs. We are actively investigating..

2. investigating Aug 17, 2026, 06:58 PM UTC

   We are seeing an increased error rate processing webhook events which will lead to delays in adoption jobs. We are actively investigating.

3. resolved Aug 17, 2026, 07:00 PM UTC

   This incident has been resolved.

4. resolved Aug 17, 2026, 07:00 PM UTC

   This incident has been resolved..

Read the full incident report →

Started Aug 17, 2026, 01:43 PM UTC· resolved Aug 17, 2026, 08:40 PM UTC

1. identified Aug 17, 2026, 01:43 PM UTC

   The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests..

2. identified Aug 17, 2026, 01:43 PM UTC

   The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests.

3. identified Aug 17, 2026, 01:46 PM UTC

   Upstream incident has been declared: We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed. We are monitoring and are looking at potential mitigations..

4. identified Aug 17, 2026, 01:46 PM UTC

   Upstream incident has been declared: We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed. We are monitoring and are looking at potential mitigations.

5. monitoring Aug 17, 2026, 04:56 PM UTC

   We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery..

6. monitoring Aug 17, 2026, 04:56 PM UTC

   We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery.

7. monitoring Aug 17, 2026, 05:52 PM UTC

   We are still seeing intermitting GitHub API errors at a low rate. .

8. monitoring Aug 17, 2026, 05:52 PM UTC

   We are still seeing intermitting GitHub API errors at a low rate.

9. monitoring Aug 17, 2026, 07:32 PM UTC

   We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage..

10. monitoring Aug 17, 2026, 07:32 PM UTC

    We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage.

11. resolved Aug 17, 2026, 08:40 PM UTC

    This incident has been resolved..

12. resolved Aug 17, 2026, 08:40 PM UTC

    This incident has been resolved.

Read the full incident report →

Started Aug 13, 2026, 11:40 AM UTC· resolved Aug 14, 2026, 06:14 PM UTC

1. investigating Aug 13, 2026, 11:40 AM UTC

   Our cloud provider is experiencing a thermal event in their us-west datacenter that is affecting caching in the region. Sticky Disk, Actions Cache, Docker Container Cache, and Bazel Build Caching are impacted: cache operations may be slow or unavailable which may result in some builds running slower than usual..

2. investigating Aug 13, 2026, 11:40 AM UTC

   Our cloud provider is experiencing a thermal event in their us-west datacenter that is affecting caching in the region. Sticky Disk, Actions Cache, Docker Container Cache, and Bazel Build Caching are impacted: cache operations may be slow or unavailable, so some builds may run slower than usual.

3. identified Aug 13, 2026, 12:06 PM UTC

   Caching performance in US West is currently degraded. Our upstream provider has lost cooling power in their data center, leading to failure of some storage systems in US West. As a result jobs may take longer to complete. We are taking action to mitigate the issue..

4. identified Aug 13, 2026, 12:06 PM UTC

   Caching performance in US West is currently degraded. Our upstream provider has lost cooling power in their data center, leading to failure of some storage systems in US West. As a result jobs may take longer to complete. We are taking action to mitigate the issue.

5. identified Aug 13, 2026, 12:36 PM UTC

   We are currently applying mitigations for affected customers in the region..

6. identified Aug 13, 2026, 12:36 PM UTC

   We are currently applying mitigations for affected customers in the region.

7. identified Aug 13, 2026, 01:02 PM UTC

   Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions..

8. identified Aug 13, 2026, 01:02 PM UTC

   Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions.

9. identified Aug 13, 2026, 01:27 PM UTC

   Conditions at our provider's us-west facility have not yet improved. As we rebalance customers away from us-west, other regions may see slightly longer pickup times than usual as a side effect.

10. identified Aug 13, 2026, 01:27 PM UTC

    As we rebalance customers away from us-west, other regions may see slightly longer pickup times than usual as a side effect

11. identified Aug 13, 2026, 02:40 PM UTC

    Conditions at our provider's us-west facility have not yet improved. We are continuing to issue mitigations to ensure that customer impact is minimal. .

12. identified Aug 13, 2026, 02:40 PM UTC

    Conditions at our provider's us-west facility have not yet improved. We are continuing to issue mitigations to ensure that customer impact is minimal.

13. identified Aug 13, 2026, 03:11 PM UTC

    Conditions in our us-west region have not yet improved. We're actively working with our datacenter provider on the issue, and will provide updates as they occur. We are also manually rebalancing traffic out of the us-west region to aid with recovery. Github have also declared an incident affecting webhooks - which may cause some job adoption delays.

14. identified Aug 13, 2026, 03:11 PM UTC

    Conditions in our us-west region have not yet improved. We're actively working with our datacenter provider on the issue, and will provide updates as they occur. We are also manually rebalancing traffic out of the us-west region to aid with recovery. Github have also declared an incident affecting webhooks - which may cause some job adoption delays

15. identified Aug 13, 2026, 04:32 PM UTC

    Our datacenter provider has begun deploying mitigations at the affected us-west facility, and GitHub has resolved its separate webhooks incident, though some knock-on delays may persist while affected jobs requeue. The us-west region remains heavily affected and with capacity reduced while we rebalance traffic, jobs in all regions may take longer than usual to start..

16. identified Aug 13, 2026, 04:32 PM UTC

    Our datacenter provider has begun deploying mitigations at the affected us-west facility, and GitHub has resolved its separate webhooks incident, though some knock-on delays may persist while affected jobs requeue. The us-west region remains heavily affected and with capacity reduced while we rebalance traffic, jobs in all regions may take longer than usual to start.

17. identified Aug 13, 2026, 05:00 PM UTC

    Our provider has partially restored cooling at the us-west facility and expects to reach safe operating temperatures within the next few hours, at which point servers will be powered back on in phases. In the meantime jobs in all regions may still take significantly longer than usual to start. We will provide another update within the next hour..

18. identified Aug 13, 2026, 05:00 PM UTC

    Our provider has partially restored cooling at the us-west facility and expects to reach safe operating temperatures within the next few hours, at which point servers will be powered back on in phases. In the meantime jobs in all regions may still take significantly longer than usual to start. We will provide another update within the next hour.

19. identified Aug 13, 2026, 05:42 PM UTC

    Our provider is continuing to restore cooling at the us-west facility; temperatures have not yet reached safe levels for servers to be powered back on, and we are staged to begin restoring runner capacity as soon as they are. The us-west region remains in a major outage, and jobs in all regions may still take longer than usual to start. We will provide another update within the next hour..

20. identified Aug 13, 2026, 05:42 PM UTC

    Our provider is continuing to restore cooling at the us-west facility; temperatures have not yet reached safe levels for servers to be powered back on, and we are staged to begin restoring runner capacity as soon as they are. The us-west region remains in a major outage, and jobs in all regions may still take longer than usual to start. We will provide another update within the next hour.

21. identified Aug 13, 2026, 06:26 PM UTC

    Our provider has begun powering servers back on at the us-west facility, and a portion of our us-west capacity is back online. We have started moving some customers back to us-west to spread load across regions. Jobs in all regions may still take longer than usual to start..

22. identified Aug 13, 2026, 06:26 PM UTC

    Our provider has begun powering servers back on at the us-west facility, and a portion of our us-west capacity is back online. We have started moving some customers back to us-west to spread load across regions. Jobs in all regions may still take longer than usual to start.

23. identified Aug 13, 2026, 07:15 PM UTC

    Our provider is continuing to bring us-west servers back online, and we are gradually moving customers back to us-west as capacity returns. The backlog has not yet cleared, so jobs in all regions may still take longer than usual to start. We will provide another update within the next hour..

24. identified Aug 13, 2026, 07:15 PM UTC

    Our provider is continuing to bring us-west servers back online, and we are gradually moving customers back to us-west as capacity returns. The backlog has not yet cleared, so jobs in all regions may still take longer than usual to start. We will provide another update within the next hour.

25. identified Aug 13, 2026, 07:59 PM UTC

    More us-west capacity has come back online and queues in the region are steadily draining as we move customers back. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage portion of the facility, and because we shifted us-west traffic to other regions earlier today, jobs elsewhere, particularly in us-east, may still take longer than usual to start. We will provide another update within the next hour..

26. identified Aug 13, 2026, 07:59 PM UTC

    More us-west capacity has come back online and queues in the region are steadily draining as we move customers back. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage portion of the facility, and because we shifted us-west traffic to other regions earlier today, jobs elsewhere, particularly in us-east, may still take longer than usual to start. We will provide another update within the next hour.

27. identified Aug 13, 2026, 08:44 PM UTC

    Queues in us-west have come down substantially as restored capacity comes online and we continue moving customers back. Cache operations, including sticky disks, remain unavailable in us-west, and jobs in us-east may still take significantly longer than usual to start while we work through the remaining backlog. We will provide another update within the next hour..

28. identified Aug 13, 2026, 08:44 PM UTC

    Queues in us-west have come down substantially as restored capacity comes online and we continue moving customers back. Cache operations, including sticky disks, remain unavailable in us-west, and jobs in us-east may still take significantly longer than usual to start while we work through the remaining backlog. We will provide another update within the next hour.

29. identified Aug 13, 2026, 09:48 PM UTC

    Queue times in us-west have returned to near-normal levels, while us-east, eu-west, and eu-central are still working through their remaining backlogs. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage systems there. We will provide another update within the next hour..

30. identified Aug 13, 2026, 09:48 PM UTC

    Queue times in us-west have returned to near-normal levels, while us-east, eu-west, and eu-central are still working through their remaining backlogs. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage systems there. We will provide another update within the next hour.

31. identified Aug 13, 2026, 10:54 PM UTC

    Queue times are improving across all regions as capacity comes back online, but jobs may still take longer than usual to start. Cache operations in us-west, including sticky disks, remain unavailable while our provider restores the storage systems, and jobs on our largest runner sizes may see the longest delays. We will provide another update within the next hour..

32. identified Aug 13, 2026, 10:54 PM UTC

    Queue times are improving across all regions as capacity comes back online, but jobs may still take longer than usual to start. Cache operations in us-west, including sticky disks, remain unavailable while our provider restores the storage systems, and jobs on our largest runner sizes may see the longest delays. We will provide another update within the next hour.

33. identified Aug 14, 2026, 12:00 AM UTC

    Runner capacity in us-west continues to recover and queues in all regions are draining. Our provider has restored the access we need to begin bringing our us-west storage systems back online. Cache operations, including sticky disks, remain unavailable while that work completes, and jobs may still take longer than usual to start..

34. identified Aug 14, 2026, 12:00 AM UTC

    Runner capacity in us-west continues to recover and queues in all regions are draining. Our provider has restored the access we need to begin bringing our us-west storage systems back online. Cache operations, including sticky disks, remain unavailable while that work completes, and jobs may still take longer than usual to start.

35. monitoring Aug 14, 2026, 01:15 AM UTC

    Queue times for all runner sizes have returned to normal in all regions, and we are monitoring closely while eu-west clears the last of its backlog. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour..

36. monitoring Aug 14, 2026, 01:15 AM UTC

    Queue times for all runner sizes have returned to normal in all regions, and we are monitoring closely while eu-west clears the last of its backlog. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour.

37. monitoring Aug 14, 2026, 02:08 AM UTC

    Job queues have fully recovered, and runners in all regions are operating normally for all runner sizes. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour..

38. monitoring Aug 14, 2026, 02:08 AM UTC

    Job queues have fully recovered, and runners in all regions are operating normally for all runner sizes. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour.

39. monitoring Aug 14, 2026, 03:06 AM UTC

    Our us-west compute provider has restored power and cooling in their datacenter and the majority of our capacity has returned. Github Actions caching has been reenabled in the region but we are still working to restore Sticky Disks, Docker Container caching, and Incremental Docker Builders. We will post an update once these components are restored..

40. monitoring Aug 14, 2026, 03:06 AM UTC

    Our provider has restored power and cooling in us-west, our capacity has returned. Cache operations, including sticky disks, remain unavailable in us-west while we complete storage recovery, and we will post an update when they are restored.

41. monitoring Aug 14, 2026, 03:26 AM UTC

    We are noticing some issues with the storage cluster backing the Github Actions cache after it was restored. We are working on restoring this alongside the storage cluster backing Sticky Disks, Incremental Docker Builders, and Docker Container Caching.

42. monitoring Aug 14, 2026, 03:26 AM UTC

    We are noticing some issues with the storage cluster backing the Github Actions cache after it was restored. We are working on restoring this alongside the storage cluster backing Sticky Disks, Incremental Docker Builders, and Docker Container Caching..

43. monitoring Aug 14, 2026, 04:09 AM UTC

    We are still working with our compute provider to recover the storage clusters..

44. monitoring Aug 14, 2026, 04:09 AM UTC

    We are still working with our compute provider to recover the storage clusters.

45. monitoring Aug 14, 2026, 06:06 AM UTC

    We have restored the Github Actions Cache in us-west. We are still working with our compute provider to restore our remaining storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders..

46. monitoring Aug 14, 2026, 06:06 AM UTC

    We have restored the Github Actions Cache in us-west. We are still working with our compute provider to restore our remaining storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders

47. monitoring Aug 14, 2026, 10:57 AM UTC

    We are continuing to work with our compute provider to restore the storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west. This requires hands-on recovery work by our provider's team and may take a few more hours to fully restore. Workflows using the affected features in us-west may see slow or stalled Docker builds in the meantime..

48. monitoring Aug 14, 2026, 10:57 AM UTC

    We are continuing to work with our compute provider to restore the storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west. This requires hands-on recovery work by our provider's team and may take a few more hours to fully restore. Workflows using the affected features in us-west may see slow or stalled Docker builds in the meantime.

49. monitoring Aug 14, 2026, 03:02 PM UTC

    The storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west has been restored, and disks are mounting and operating normally. As part of the recovery, a small amount of recently written cache data may need to be rebuilt, so some Docker builds may run slower over their first few runs while caches rehydrate. We are monitoring closely..

50. monitoring Aug 14, 2026, 03:02 PM UTC

    The storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west has been restored, and disks are mounting and operating normally. As part of the recovery, a small amount of recently written cache data may need to be rebuilt, so some Docker builds may run slower over their first few runs while caches rehydrate. We are monitoring closely.

51. monitoring Aug 14, 2026, 04:16 PM UTC

    All services have been restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. We are continuing to monitor the stability of the recovered storage cluster as it ramps back up with traffic, and some builds may run slower on their first runs while recently written cache data rebuilds. We will post a final update once we have confirmed stability..

52. monitoring Aug 14, 2026, 04:16 PM UTC

    All services have been restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. We are continuing to monitor the stability of the recovered storage cluster as it ramps back up with traffic, and some builds may run slower on their first runs while recently written cache data rebuilds. We will post a final update once we have confirmed stability.

53. resolved Aug 14, 2026, 06:14 PM UTC

    All services have been fully restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. A small amount of recently written cache data could not be recovered during storage repair, so some builds may run slower on their first runs while caches rebuild. This incident is now resolved, and we will publish a detailed post-incident report in the coming days..

54. resolved Aug 14, 2026, 06:14 PM UTC

    All services have been fully restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. A small amount of recently written cache data could not be recovered during storage repair, so some builds may run slower on their first runs while caches rebuild. This incident is now resolved, and we will publish a detailed post-incident report in the coming days.

Read the full incident report →

Started Aug 12, 2026, 06:41 PM UTC· resolved Aug 13, 2026, 01:20 AM UTC

1. investigating Aug 12, 2026, 06:41 PM UTC

   We are investigating reports of socket hangups when attempting to fetch GitHub release assets. There are not any current upstream incidents, but we are continuing to investigate..

2. investigating Aug 12, 2026, 06:41 PM UTC

   We are investigating reports of socket hangups when attempting to fetch GitHub release assets. There are not any current upstream incidents, but we are continuing to investigate.

3. identified Aug 12, 2026, 07:25 PM UTC

   We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures have been isolated to GitHub upstream, and we are monitoring closely while the issue persists. We have observed multiple public reports that GitHub is experiencing a partial outage..

4. identified Aug 12, 2026, 07:25 PM UTC

   We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures have been isolated to GitHub upstream, and we are monitoring closely while the issue persists. We have observed multiple public reports that GitHub is experiencing a partial outage.

5. identified Aug 12, 2026, 07:55 PM UTC

   We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures remain isolated to GitHub upstream, and we are continuing to monitor for recovery; other GitHub interactions, including repository checkout, remain unaffected. We will provide another update within the next 30 minutes..

6. identified Aug 12, 2026, 07:55 PM UTC

   We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures remain isolated to GitHub upstream, and we are continuing to monitor for recovery; other GitHub interactions, including repository checkout, remain unaffected. We will provide another update within the next 30 minutes.

7. identified Aug 12, 2026, 08:23 PM UTC

   We have isolated the cause to a subset of GitHub's edge servers and verified the failures occur from networks outside Blacksmith as well. Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs is the most effective workaround until the upstream issue is resolved. We will provide another update within the next 30 minutes..

8. identified Aug 12, 2026, 08:23 PM UTC

   We have isolated the cause to a subset of GitHub's edge servers and verified the failures occur from networks outside Blacksmith as well. Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs is the most effective workaround until the upstream issue is resolved. We will provide another update within the next 30 minutes.

9. identified Aug 12, 2026, 08:52 PM UTC

   Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs remains the most effective workaround. The failures remain isolated to a subset of GitHub's edge servers upstream of our infrastructure, and we are continuing to monitor for recovery..

10. identified Aug 12, 2026, 08:52 PM UTC

    Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs remains the most effective workaround. The failures remain isolated to a subset of GitHub's edge servers upstream of our infrastructure, and we are continuing to monitor for recovery.

11. identified Aug 12, 2026, 09:31 PM UTC

    Workflows downloading GitHub release assets remain affected across all regions because of an upstream issue at GitHub, and retrying failed jobs remains the most effective workaround..

12. identified Aug 12, 2026, 09:31 PM UTC

    Workflows downloading GitHub release assets remain affected across all regions because of an upstream issue at GitHub, and retrying failed jobs remains the most effective workaround.

13. identified Aug 12, 2026, 10:03 PM UTC

    GitHub has confirmed an incident affecting release asset downloads, consistent with our findings (githubstatus.com/incidents/lsvy8xsf0gxv). Workflows downloading GitHub release assets remain affected across all regions while GitHub works toward mitigation, and retrying failed jobs remains the most effective workaround. .

14. identified Aug 12, 2026, 10:03 PM UTC

    GitHub has confirmed an incident affecting release asset downloads, consistent with our findings (githubstatus.com/incidents/lsvy8xsf0gxv). Workflows downloading GitHub release assets remain affected across all regions while GitHub works toward mitigation, and retrying failed jobs remains the most effective workaround.

15. monitoring Aug 12, 2026, 10:53 PM UTC

    We implemented a fix and are currently monitoring the result.

16. monitoring Aug 12, 2026, 11:06 PM UTC

    GitHub has implemented a fix and we are beginning to see recovery. We will continue to monitor the issue. .

17. monitoring Aug 12, 2026, 11:06 PM UTC

    GitHub has implemented a fix and we are beginning to see recovery. We will continue to monitor the issue.

18. resolved Aug 13, 2026, 01:20 AM UTC

    This incident has been resolved..

19. resolved Aug 13, 2026, 01:20 AM UTC

    This incident has been resolved.

Read the full incident report →

Started Aug 12, 2026, 03:30 AM UTC· resolved Aug 12, 2026, 11:50 AM UTC

1. investigating Aug 12, 2026, 03:30 AM UTC

   We are currently investigating this incident.

2. investigating Aug 12, 2026, 03:30 AM UTC

   We are currently investigating this incident..

3. identified Aug 12, 2026, 09:10 AM UTC

   We have identified the affected mirror and are implementing a fix.

4. identified Aug 12, 2026, 09:10 AM UTC

   We have identified the affected mirror and are implementing a fix

5. identified Aug 12, 2026, 10:33 AM UTC

   We are currently deploying a mitigation.

6. identified Aug 12, 2026, 10:33 AM UTC

   We are currently deploying a mitigation

7. monitoring Aug 12, 2026, 11:26 AM UTC

   We implemented a fix and are seeing improvements. We are continuing to monitor the result..

8. monitoring Aug 12, 2026, 11:26 AM UTC

   We implemented a fix and are seeing improvements. We are continuing to monitor the result.

9. resolved Aug 12, 2026, 11:50 AM UTC

   This incident has been resolved..

10. resolved Aug 12, 2026, 11:50 AM UTC

    This incident has been resolved.

Read the full incident report →

Started Aug 11, 2026, 07:00 PM UTC· resolved Aug 11, 2026, 08:00 PM UTC

1. investigating Aug 11, 2026, 07:00 PM UTC

   We are observing certain customers experiencing higher than baseline cache miss rates..

2. investigating Aug 11, 2026, 07:00 PM UTC

   We are observing certain customers experiencing higher than baseline cache miss rates.

3. resolved Aug 11, 2026, 08:00 PM UTC

   We resolved the configuration error leading to the increased rate of cache misses. Cache hit rates are now at the baseline.

4. resolved Aug 11, 2026, 08:00 PM UTC

   We resolved the configuration error leading to the increased rate of cache misses. Cache hit rates are now at the baseline..

Read the full incident report →

Started Aug 10, 2026, 07:17 PM UTC· resolved Aug 10, 2026, 08:44 PM UTC

1. investigating Aug 10, 2026, 07:17 PM UTC

   We are experiencing degradation in our metrics and log ingestion services in our us-west region. We are actively investigating the issue..

2. investigating Aug 10, 2026, 07:17 PM UTC

   We are experiencing degradation in our metrics and log ingestion services in our us-west region. We are actively investigating the issue.

3. investigating Aug 10, 2026, 07:44 PM UTC

   We are continuing to investigate degraded metrics and log ingestion in our us-west region. Customers may still see metrics and logs for their jobs appear missing or delayed in the Blacksmith dashboard, while other regions remain unaffected. We will provide another update within the next 30 minutes..

4. investigating Aug 10, 2026, 07:44 PM UTC

   We are continuing to investigate degraded metrics and log ingestion in our us-west region. Customers may still see metrics and logs for their jobs appear missing or delayed in the Blacksmith dashboard, while other regions remain unaffected. We will provide another update within the next 30 minutes.

5. investigating Aug 10, 2026, 07:45 PM UTC

   We are experiencing degraded network performance in our us-west region, affecting metrics and log ingestion as well as job performance. Jobs in us-west that upload artifacts or transfer large amounts of data may run slower than normal and in some cases hit their configured timeouts and fail. Other regions are not affected, and we are actively investigating the issue..

6. investigating Aug 10, 2026, 07:45 PM UTC

   We are experiencing degraded network performance in our us-west region, affecting metrics and log ingestion as well as job performance. Jobs in us-west that upload artifacts or transfer large amounts of data may run slower than normal and in some cases hit their configured timeouts and fail. Other regions are not affected, and we are actively investigating the issue.

7. monitoring Aug 10, 2026, 08:18 PM UTC

   Metrics and log ingestion in our us-west region is recovering, and job performance in the region has returned to normal. We are monitoring to confirm the recovery holds and are continuing to investigate the underlying cause. We will provide an update shortly..

8. monitoring Aug 10, 2026, 08:18 PM UTC

   Metrics and log ingestion in our us-west region is recovering, and job performance in the region has returned to normal. We are monitoring to confirm the recovery holds and are continuing to investigate the underlying cause. We will provide an update shortly.

9. resolved Aug 10, 2026, 08:44 PM UTC

   This incident is resolved, with job performance and the ingestion of metrics and logs in our us-west region stable for the past 30 minutes. Jobs that failed or timed out during the incident can be safely re-run. .

10. resolved Aug 10, 2026, 08:44 PM UTC

    This incident is resolved, with job performance and the ingestion of metrics and logs in our us-west region stable for the past 30 minutes. Jobs that failed or timed out during the incident can be safely re-run.

Read the full incident report →

Started Aug 06, 2026, 03:30 PM UTC· resolved Aug 07, 2026, 12:57 AM UTC

1. monitoring Aug 06, 2026, 03:30 PM UTC

   GitHub has reported degraded performance for Actions. Jobs may be delayed to start, or fail during the set up step. We are monitoring this incident. .

2. monitoring Aug 06, 2026, 03:30 PM UTC

   Github has reported degraded performance for Actions, jobs may take a moment to be adopted. We are monitoring this incident.

3. resolved Aug 07, 2026, 12:57 AM UTC

   GitHub job success and adoption rates are now back at normal levels. Jobs that were not adopted during the incident are being requeued by our team and should be picked up shortly..

4. resolved Aug 07, 2026, 12:57 AM UTC

   GitHub job success and adoption rates are now back at normal levels. Jobs that were not adopted during the incident are being requeued by our team and should be picked up shortly.

Read the full incident report →

Started Aug 06, 2026, 01:18 AM UTC· resolved Aug 06, 2026, 02:53 AM UTC

1. investigating Aug 06, 2026, 01:18 AM UTC

   We're currently experiencing an outage of our control plane. GitHub job adoption and execution are affected, as well as dashboard access..

2. investigating Aug 06, 2026, 01:18 AM UTC

   We're currently experiencing an outage of our control plane. GitHub job adoption and execution are affected, as well as dashboard access.

3. identified Aug 06, 2026, 01:36 AM UTC

   We've mitigated the root cause and jobs are resuming to run. Some jobs may still be delayed to start as we catch up with the job backlog..

4. identified Aug 06, 2026, 01:36 AM UTC

   We've mitigated the root cause and jobs are resuming to run. Some jobs may still be delayed to start as we catch up with the job backlog.

5. monitoring Aug 06, 2026, 01:51 AM UTC

   Jobs adoption has recovered and are operational..

6. monitoring Aug 06, 2026, 01:51 AM UTC

   Jobs adoption has recovered and are operational.

7. monitoring Aug 06, 2026, 02:11 AM UTC

   There is still a delay in job adoption times as we continue to recover..

8. monitoring Aug 06, 2026, 02:11 AM UTC

   There is still a delay in job adoption times as we continue to recover.

9. monitoring Aug 06, 2026, 02:45 AM UTC

   Job adoption is now fully operational..

10. monitoring Aug 06, 2026, 02:45 AM UTC

    Job adoption is now fully operational.

11. resolved Aug 06, 2026, 02:53 AM UTC

    This incident has been resolved..

12. resolved Aug 06, 2026, 02:53 AM UTC

    This incident has been resolved.

Read the full incident report →

Started Aug 05, 2026, 10:45 PM UTC· resolved Aug 05, 2026, 11:37 PM UTC

1. investigating Aug 05, 2026, 10:45 PM UTC

   We are seeing elevated error rates for Sticky Disk and Actions Cache requests. We are investigating.

2. monitoring Aug 05, 2026, 10:55 PM UTC

   We implemented a fix and are currently monitoring the result.

3. monitoring Aug 05, 2026, 11:30 PM UTC

   We are still monitoring recovery.

4. resolved Aug 05, 2026, 11:37 PM UTC

   This incident has been resolved.

Read the full incident report →