Blacksmith incident

Upstream Ubuntu package mirror outage affecting apt

Minor Resolved

Blacksmith experienced a minor incident on September 11, 2026, lasting 9h 29m. The incident has been resolved; the full update timeline is below.

Started
Sep 11, 2026, 05:30 AM UTC
Resolved
Sep 11, 2026, 02:59 PM UTC
Duration
9h 29m
Detected by Pingoru
Sep 11, 2026, 05:30 AM UTC

Update timeline

  1. monitoring Sep 11, 2026, 05:30 AM UTC

    Since 06:30 UTC Canonical has had three outages affecting [archive.ubuntu.com](http://archive.ubuntu.com), [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.

  2. monitoring Sep 11, 2026, 02:21 PM UTC

    We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs.

  3. resolved Sep 11, 2026, 02:59 PM UTC

    This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future.