Jenkins Outage History

The capacity problem has resolved by Azure: we saw builds being handled again around 16:00 UTC.

Since Wednesday 1st July 2026 at 13:00 UTC, we are unable to execute pipelines on trusted.ci.jenkins.io and infra.ci.jenkins.io. Azure reports capacity issues in US East 1 and they are unable to fulfill our virtual machines creation requests.

The cluster and all its services were up and running at 10:02am UTC after many mitigation and operations from the Jenkins Infra team.

At 08:50am UTC, all services were down due to internal AKS outage (combination of network issue and Azure forbidding us to spin up new VMs).

We tried to apply mitigations between 08:15am UTC and 08:45am UTC without any success (draining AKS nodes to restart services on fresh VMs).

Starting on Tuesday 16 June 2026 at 07:30am UTC, the cluster publick8s hosting our public web services such as updates.jenkins.io, get.jenkins.io, LDAP and many others started to drop incoming TCP connections.

JIRA is back.

The JIRA instance at issues.jenkins.io, managed by the Linux Foundation team, is down since at 13:20 (UTC).

Jenkins website and some other services are down, currently investigating.

On Wednesday 28 January 2026 at 18h15 UTC, ci.jenkins.io will require authentication with a Jenkins LDAP account to allow reading pages.

CloudFlare maintenance is finished: we’ve enabled the East America mirror as it is now responding as expected.

CloudFlare is performing a maintenance (https://www.cloudflarestatus.com/incidents/y5xbgk3fh95y) with re-routed traffic. Since 07:30 UTC, our monitoring detected a lot of timeouts for the Update Center requests targeted at our East US mirror due to the re-routing latency.

The ci.jenkins.io controller is back to a normal response time. The AI crawler has stopped sending requests. We had to restart the controller once per hour to sustain the high rate of requests.

–> ci.jenkins.io is failing with Out Of Memory errors, we’ve restarted it. We don’t know yet the exact cause but we’re suspecting AI crawlers.

The outage seems has been resolved at 00:05am on the 30 October.

Azure is facing a global outage (Tracking ID: YKYN-BWZ at https://azure.status.microsoft/en-us/status/history/) which prevents access to the Azure Portal and to the Azure Identity Management API.

The error has been identified and fixed (token regeneration). Service is back to normal.

We are encountering an outage on incrementals.jenkins.io, currently investigating.

The outage seems resolved https://github.com/jenkins-infra/helpdesk/issues/4827

we are encountering an outage on get.jenkins.io and updates.jenkins.io our publick8s cluster. We are seeing problems that azure doesn’t acknowledge, probably related to an Azure outage.

The migration went well, all tests are green.

Thursday October 02 2025, usage.jenkins.io VM will be migrated from AWS to Digital Ocean as per https://github.com/jenkins-infra/helpdesk/issues/1626.

The ci.jenkins.io controller is now up and running, the brownout is done, we will have to work around permission for the next brownout.

A large set of unhelpful requests are overloading ci.jenkins.io. We are in a brownout. We are testing that disallowing anonymous access resolves the performance issue.