Depot Outage History

Status: Identified Identified the issue and working on a fix. Affected components us-east-1 (Degraded performance) Depot CI (Operational)

Status: Identified We have identified the cause as an outage of our primary database Affected components eu-central-1 (Full outage) API (Full outage) Depot Cache (Full outage) Container Registry (Full outage) Depot CI (Full outage) us-east-1 (Full outage) Dashboard (Full outage) us-east-1 (Full outage)

Status: Identified We have worked with our database provider to restore database connectivity - CI jobs and container builds are operational and are now processing the backlog of jobs. Affected components eu-central-1 (Degraded performance) API (Operational) Depot Cache (Degraded performance) Container Registry (Degraded performance) Depot CI (Degraded performance) us-east-1 (Degraded performance) Dashboard (Operational) us-east-1 (Degraded performance)

Status: Identified We are continuing to see capacity issues across us-east-1 and us-east-2 inside of AWS. While this is ongoing, we are rerouting jobs to AZs in regions with capacity. We are in contact with AWS to get additional capacity in the impacted regions released. This degraded performance is only impacting GitHub Actions runners to date. Affected components us-east-1 (Degraded performance)

Status: Identified We have identified a capacity issue affecting some GitHub Actions jobs in us-east-1 and have deployed a routing configuration change to address. Affected components us-east-1 (Degraded performance)

Status: Identified We are seeing a recurrence of upstream capacity issues and elevated queue times for GitHub Actions jobs. We are closely monitoring and working on additional mitigations. Affected components us-east-1 (Degraded performance)

Status: Identified We are continuing to see upstream capacity issues acquiring machines for GitHub jobs. We are actively failing over to additional regions as capacity becomes available. Affected components us-east-1 (Degraded performance)

Status: Identified Experiencing periodic 502's on various pages of the depot.dev web application. Affected components Dashboard (Degraded performance)

Status: Identified We have identified an issue causing a small percentage of x86 GitHub Actions jobs to fail to start and have deployed a mitigation. Affected components us-east-1 (Partial outage)

Status: Identified We are continuing to see reduced AWS capacity in us-east-1 and us-east-2, causing increased queue times or failures to start for some jobs. Affected components us-east-1 (Degraded performance)

Status: Identified Errors seem to be caused in part by lack of instance capacity in our primary runner instance pool, we are activating a failover instance pool to unblock jobs. Affected components us-east-1 (Partial outage)

Status: Identified Failover capacity is online and job throughput is increasing. Affected components us-east-1 (Degraded performance)

Status: Identified AWS are experiencing broad capacity constraints around our preferred instance type, we are activating a fallback to secondary instance type to address. Affected components us-east-1 (Degraded performance)

Status: Identified We are working with our database provider to deploy a mitigation for the increased read-replica lag. Affected components Dashboard (Degraded performance) API (Degraded performance)

Status: Identified We have identified the issue and are working on a fix. Affected components us-east-1 (Partial outage)

Status: Identified The increased delay is related to decreased AWS capacity in a single zone, the Depot control plane is routing traffic around the affected zone. Affected components us-east-1 (Degraded performance)

Status: Identified We have identified a GitHub API issue causing this delay. Jobs are starting to process now, we have deployed additional capacity to handle the incoming jobs. Affected components us-east-1 (Degraded performance)