Namespace Outage History

Status: Identified GitHub is tracking this at https://www.githubstatus.com/incidents/8tzbpcnl75jp.

Status: Identified We're investigating degraded performance of registry operations in our Western Europe region. Affected components Western Europe (Degraded performance) Container Registry (Degraded performance)

Status: Identified We identified the issue and applied a mitigation, queue time starts decreasing. Impact was increased queue time, up to 15 minutes of queue time for a small set of jobs. Affected components GitHub Runners (Operational)

Status: Identified We identified and mitigated an issue resulting in many instances seeing delayed start. The mitigation is holding and we're seeing first signs of recovery, but are processing a large backlog of jobs. Affected components GitHub Runners (Degraded performance)

Status: Identified We're seeing delays in our log processing pipeline, and are working on expanding capacity to improve this. Instance logs for some instances will take longer to show up.

Status: Identified GitHub is rejecting a subset of job acquisitions with MissingKey. There is no corresponding upstream incident yet.

Status: Identified We're seeing increased queue time for a subset of jobs, the team is working on mitigation. Affected components GitHub Runners (Degraded performance)

Status: Identified We're seeing severely degraded checkout performance, we're investigating the cause. This causes to increased runtime and concurrency exhaustion, leading to increased queue time. Affected components GitHub Runners (Degraded performance)

Status: Identified Checkout performance continues to be degraded, the team is root-causing the issue and working on mitigation options. Affected components GitHub Runners (Degraded performance) Northern Virginia, US (Degraded performance)

Status: Identified We continue seeing degraded networking performance, the team is working on establishing an alternative transit path. Affected components GitHub Runners (Degraded performance) Northern Virginia, US (Degraded performance)

Status: Identified We see partial recovery after mitigations were applied, but networking continues to show degradation in the high percentiles. The team is working on deploying additional capacity to work towards a full recovery. Affected components GitHub Runners (Operational) Northern Virginia, US (Degraded performance)

Status: Identified A fix has been applied. The dashboard has recovered. A subset of jobs is still seeing elevated queue time. Affected components Dashboard (Operational) Northern Virginia, US (Degraded performance)

Status: Identified We're observing degraded registry and artifact storage performance in one of our compute shards. Affected components Container Registry (Degraded performance) Artifacts (Degraded performance)

Status: Identified The cause has been identified, and a fix is being released. Affected components SAML SSO (Full outage)

Status: Identified We've applied an initial mitigation and are seeing first signs of recovery. We're processing a large the backlog of jobs. Affected components GitHub Runners (Degraded performance)

Status: Identified We identified and stopped a single replica that was causing the stalls. Affected components Gradle (Degraded performance) HTTP Cache (Degraded performance) Illinois, US (Operational) Bazel (Degraded performance) Artifacts (Degraded performance) Turborepo (Degraded performance)

Status: Identified The new replica showed the same stalling pattern. We identified a networking misconfiguration that's causing the issue. Affected components Turborepo (Degraded performance) Gradle (Degraded performance) HTTP Cache (Degraded performance) Illinois, US (Operational) Bazel (Degraded performance) Artifacts (Degraded performance)

Status: Identified We are seeing elevated instance creation time in our European compute shard. Traffic is being steered to other shards. Affected components Western Europe (Degraded performance) Devboxes (Degraded performance)

Status: Identified There is elevated latency on instance creation in our European compute shard. Traffic is being steered to other shards. The team is working on a resolution. Affected components Western Europe (Degraded performance) Devboxes (Degraded performance)

Status: Identified Pulls from external registries go through a caching layer in our infrastructure, which is serving incorrect data for a small subset of images. A mitigation has been applied to restore correct behavior for failing images.

Status: Identified There is still a small set of images for which an incorrect response is served, we've identified the cause and are rolling out a fix.

Status: Identified One of our compute shards is undergoing maintenance. That may affect loading the instance list.

Status: Identified We have identified a networking issue and are working on a solution. The affected compute shards have been removed from the scheduling rotation temporarily. Affected components GitHub Runners (Degraded performance)

Status: Identified We see recovery for most customers. A subset of MacOS GitHub jobs is still being delayed and the team is root causing the problem. Affected components GitHub Runners (Degraded performance)

Status: Identified We have identified the issue and applied mitigation. We continue to observe the performance of traffic to GitHub. Affected components Northern Virginia, US (Degraded performance)