Is Blacksmith down?

We are currently observing network stalling in our US East region with runner connectivity against AWS's us-east-1 region. Uploads and downloads to and from ECR and S3 may experience occasional stalling. We are investigating.

We have identified a small subset of runners that are exhibiting this network stalling and have isolated them. We're continuing to monitor recovery.

This incident has been resolved.

Since 06:30 UTC Canonical has had three outages affecting [archive.ubuntu.com](http://archive.ubuntu.com), [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.

We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.

This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future.

We are currently experiencing a major outage with both our caching infrastructure and our dashboards. These components are affected across all regions. We are investigating the issue and will provide an update soon.

We are seeing recovery of our caching infrastructure and our dashboard. We will continue to monitor this situation closely.

This incident has been resolved.

Since approximately 12:20 UTC, some jobs in our eu-west region have been failing with the GitHub error "The self-hosted runner lost communication with the server"; other regions are unaffected. Affected jobs can be safely re-run while we continue to investigate the underlying cause.

Between approximately 12:20 and 15:15 UTC, some jobs in our eu-west region failed with the GitHub error "The self-hosted runner lost communication with the server"; other regions were unaffected. Job failures in eu-west returned to normal levels as of 15:15 UTC, and any affected jobs can be safely re-run while we continue to investigate the underlying cause.

This incident is resolved. Job failures in our eu-west region returned to normal levels at 15:15 UTC and have remained there since; any jobs that failed between approximately 12:20 and 15:15 UTC with the error "The self-hosted runner lost communication with the server" can be safely re-run.

We are investigating reports of higher latency on sticky disk operations in our us-west region. Customers running jobs in us-west may see slower incremental Docker builds, Git caching, and container caching, so affected jobs can take longer than usual to complete.

Sticky disk storage in our us-west region is experiencing higher latency; our other regions are not affected. Customers running jobs in us-west may see slower Git-cached checkouts, incremental Docker builds, and container caching. We are continuing to investigate the underlying cause and will provide another update within the next 30 minutes.

We are still working to resolve elevated latency on sticky disk storage in our us-west region; other regions are not affected. Customers running jobs in us-west may continue to see slower Git-cached checkouts, incremental Docker builds, and container caching. Our investigation into the underlying cause is ongoing and we will provide another update within the next 30 minutes.

Our us-west region storage cluster's latency and error rates have returned to baseline as of approximately 17:15 UTC, following mitigations that reduce load on the affected storage. Git-cached checkouts, incremental Docker builds, and container caching in us-west are back to normal, and we are monitoring to confirm the recovery holds while we continue to investigate the underlying cause. We will provide a final update within the next hour.

This incident is resolved. Sticky disk storage in our us-west region came under more read load than it could serve at normal latency, which slowed and produced higher error rates for Git-cached checkouts, incremental Docker builds, and container caching. We reduced and redistributed that load, and performance has been normal since approximately 17:15 UTC. Jobs that failed during the incident can be safely re-run.