Blacksmith incident
Upstream Ubuntu package mirror outage affecting apt
Blacksmith experienced a minor incident on September 11, 2026, lasting 9h 29m. The incident has been resolved; the full update timeline is below.
Update timeline
- monitoring Sep 11, 2026, 05:30 AM UTC
Since 06:30 UTC Canonical has had three outages affecting [archive.ubuntu.com](http://archive.ubuntu.com), [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.
- monitoring Sep 11, 2026, 02:21 PM UTC
We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.
- resolved Sep 11, 2026, 02:59 PM UTC
This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future.