Incident with Actions
Affected: GitHub Actions Service (GitHub → Actions)
Ubicloud Outage History
Ubicloud is up right nowUbicloud had 19 outages in the last 2 years totaling 216h 28m of downtime — averaging 0.8 incidents per month.
There were 19 Ubicloud outages since November 18, 2025 totaling 216h 28m of downtime. Each is summarised below — incident details, duration, and resolution information.
Actions Job Delays
Affected: GitHub Actions Service (GitHub → Actions)
Timeline · 10 updates
Increased provisioning times for large runner sizes (16 and 30 vCPUs)
Affected: Ubicloud Runners
Timeline · 2 updates
- investigating Oct 01, 2026, 02:37 PM UTC
We're seeing higher than usual demand for our runners. This is causing longer than usual provisioning times for larger runners (16 and 30 vCPUs). We're working to increase our capacity as quickly as possible and mitigate the issue.
- resolved Oct 01, 2026, 04:24 PM UTC
Following an initial capacity increase, provisioning times have returned to normal. We're continuing to expand our fleet to meet future demand.
Intermittent failures on Canonical's official Ubuntu apt repositories
Affected: Ubicloud Runners
Timeline · 2 updates
- monitoring Sep 11, 2026, 03:00 PM UTC
Canonical's Ubuntu archive mirrors have gone down four separate times today. Their status page (https://status.canonical.com/#/?section=spIncidentHistory) lists each incident as brief and now resolved, but we are still seeing intermittent connection failures against those mirrors, which can cause apt-get steps to hang or fail. The Azure mirrors are holding up better right now. As a workaround, you can point your apt sources at azure.archive.ubuntu.com instead of archive.ubuntu.com and security.ubuntu.com by adding this step to your workflow: - run: printf "http://azure.archive.ubuntu.com/ubuntu/\tpriority:1\nhttps://archive.ubuntu.com/ubuntu/\tpriority:2\nhttps://security.ubuntu.com/ubuntu/\tpriority:3\n" | sudo tee /etc/apt/apt-mirrors.txt This keeps the official mirrors as lower-priority fallbacks, so builds still resolve packages if Azure becomes unreachable. We are monitoring the situation closely.
- resolved Sep 11, 2026, 10:38 PM UTC
Canonical's Ubuntu apt mirrors have stabilized and we are no longer seeing failures on apt installs.
Increased provisioning times for GitHub runners
Affected: Ubicloud Runners
Timeline · 2 updates
- investigating Aug 25, 2026, 03:54 PM UTC
We're seeing higher than usual demand for our runners. This is causing longer than usual provisioning times. We're working to increase our capacity as quickly as possible and mitigate the issue.
- resolved Aug 25, 2026, 05:51 PM UTC
Provisioning times have returned to normal.
Degraded availability for GitHub runners
Affected: GitHub ActionsGitHub API Requests
Timeline · 4 updates
- investigating Jul 25, 2026, 12:36 PM UTC
Github runners availability has been degraded because of an incident in the upstream Github API. https://www.githubstatus.com/incidents/pz7g535gbs6p
- investigating Jul 25, 2026, 12:46 PM UTC
We are continuing to investigate this issue.
- monitoring Jul 25, 2026, 01:04 PM UTC
Number of errors from the upstream Github API has reduced. Although, the upstream incident is still active https://www.githubstatus.com/incidents/pz7g535gbs6p
- resolved Jul 25, 2026, 02:07 PM UTC
The runners should be back to normal. The upstream github actions API incident has been resolved.
Issues with Postgres databases
Affected: PostgreSQL
Timeline · 3 updates
- investigating Jun 27, 2026, 07:29 PM UTC
We are currently investigating issues which mostly impacts Postgres services.
- monitoring Jun 27, 2026, 08:08 PM UTC
Hetzner experienced a network outage at Falkenstein that affected some of our services: https://status.hetzner.com/incident/49567296-8543-43c1-a0e2-9755ad4bd018
- resolved Jun 27, 2026, 08:09 PM UTC
The issue should now be resolved. Please let us know if you are still experiencing any problems. We appreciate your patience while we worked to resolve this.
Service Disruption in the US region due to networking outage
Affected: ComputePostgreSQLNetworkingKubernetes
Timeline · 4 updates
- investigating Jun 10, 2026, 06:52 PM UTC
We are currently investigating and working on fixing the problem.
- investigating Jun 10, 2026, 06:56 PM UTC
We are continuing to investigate this issue.
- investigating Jun 10, 2026, 07:56 PM UTC
We experience problems in a subset of the underlying hardware, we are working with the provider to address these issues.
- resolved Jun 11, 2026, 12:11 AM UTC
All customers have been migrated from affected hardware. If you notice anything out of place, email [email protected]
Degraded git checkouts on GitHub Actions runners (upstream GitHub)
Affected: GitHub Actions
Timeline · 5 updates
- investigating May 26, 2026, 11:02 AM UTC
We are investigating reports of slow or failing git checkout and git fetch steps on Ubicloud GitHub Actions runners. Jobs may stall, run very slowly, or time out during the checkout step. We are working to confirm the source.
- identified May 26, 2026, 11:06 AM UTC
We have traced this to an upstream GitHub issue affecting git data transfer. Connections to GitHub establish normally (DNS, TCP, and TLS are healthy), but transfers stall mid-stream, producing very slow clone and fetch speeds, HTTP/2 stream cancellations, or checkout steps that hang for several minutes. This is occurring across multiple regions and is not specific to Ubicloud infrastructure. The behavior is being reported and tracked by others in the GitHub community here: https://github.com/orgs/community/discussions/196638. We will continue to monitor and update as GitHub addresses the underlying problem. If you need to mitigate in the meantime, the following can help on affected jobs: shallow clones (fetch-depth: 1), forcing HTTP/1.1 (git config --global http.version HTTP/1.1), and adding retry logic around the checkout step.
- identified May 26, 2026, 11:37 AM UTC
GitHub has a separate, major outage https://www.githubstatus.com/incidents/gnftqj9htp0g
- identified May 29, 2026, 08:04 AM UTC
We are continuing to work on a fix for this issue.
- resolved Jun 02, 2026, 06:57 PM UTC
GitHub confirmed that they resolved the issue on their edge servers.
Degraged GitHub Webhooks
Affected: GitHub Webhooks
Timeline · 2 updates
- identified May 19, 2026, 03:05 PM UTC
At approximately 14:55 UTC, we observed a drop in incoming GitHub webhooks as GitHub temporarily stopped sending events. Delivery has since partially recovered. Our system automatically detects missing webhooks every 5 minutes and self-heals by reconciling the missed events. As a result, you may experience longer-than-usual job assignment times until delivery fully stabilizes. We are continuing to monitor the situation.
- resolved May 19, 2026, 05:31 PM UTC
This incident has been resolved.
Networking issue impacting multiple services
Affected: PostgreSQLNetworking
Timeline · 3 updates
- investigating Mar 20, 2026, 10:34 AM UTC
We are currently investigating issues which mostly impacts Postgres services.
- investigating Mar 20, 2026, 10:35 AM UTC
We are continuing to investigate this issue.
- resolved Mar 20, 2026, 11:55 AM UTC
This incident has been resolved.
High job provisioning delays for arm64 GitHub runners
Affected: Ubicloud Runners
Timeline · 2 updates
- identified Mar 06, 2026, 01:58 PM UTC
Due to an unexpected spike in arm64 demand, there is currently a delay in provisioning arm64 runners. We are working to increase our capacity as quickly as possible to meet the demand and mitigate the issue.
- resolved Mar 06, 2026, 02:56 PM UTC
This incident has been resolved.
Occassional DNS Resolution Failures
Affected: ComputePostgreSQLNetworkingKubernetes
Timeline · 2 updates
- identified Feb 25, 2026, 09:37 PM UTC
DNS queries can fail sporadically, linked to a UDP-related fault at Hetzner https://status.hetzner.com/incident/5e901309-0873-42af-90c5-84f38304f592 We're monitoring and looking to work around this issue.
- resolved Feb 25, 2026, 10:55 PM UTC
Caused by a DDOS attack against Hetzner and its mitigation routines. We will be diversifying our DNS upstreams in the future.
Erratic restarts on some VMs and Databases
Affected: ComputePostgreSQLKubernetes
Timeline · 2 updates
- monitoring Feb 13, 2026, 08:13 PM UTC
Some systems have been restarting erratically due to a recent code change. We backed this out and are recovering.
- resolved Feb 13, 2026, 08:16 PM UTC
All systems operational again. Write to [email protected] about anything unexpected.
Postgres DNS resolution issues
Affected: PostgreSQL
Timeline · 5 updates
- investigating Feb 06, 2026, 08:33 PM UTC
There is a defect in Postgres that causes unbound DNS records. Negative caching can exacerbate the incident after our fix. Databases affected: many at the outset, some now. Diagnosing the last records.
- monitoring Feb 06, 2026, 09:16 PM UTC
We've hotfixed the remaining records. Negative caching of DNS can slow recovery, depending on the upstream DNS server. We will remain in contact with all affected customer until all symptoms are resolved.
- identified Feb 06, 2026, 09:35 PM UTC
Some records are proving more difficult to address than we expected.
- monitoring Feb 06, 2026, 09:41 PM UTC
We've identified this problem, and believe negative caching of DNS records to be all that remains. We are going to continue to check for at least thirty minutes, the longest time reported for negative caching so far.
- resolved Feb 06, 2026, 10:13 PM UTC
Cross checking DNS records has been successful for some time. Customers report negative caches have since expired. Likely cause: race condition that caused misapplication of DNS records that was exposed under higher parallelism. We have to analyze it and patch it in the coming days.
Intermittent Kubernetes Networking Outages
Timeline · 1 update
- resolved Feb 09, 2026, 10:06 AM UTC
We are observed temporary networking issues impacting mesh stability and external connectivity.
A host in eu-central-h1 is experiencing a hardware failure
Affected: ComputePostgreSQL
Timeline · 2 updates
- identified Jan 19, 2026, 02:26 PM UTC
Instances running on this host are currently unreachable. We are actively working to mitigate the issue and restore services. We will keep you updated as we make progress.
- resolved Jan 19, 2026, 03:13 PM UTC
Our hardware provider has replaced the failed hardware, and all systems are now fully operational.
Increased job queue times in Ubicloud runners
Affected: Ubicloud Runners
Timeline · 3 updates
- investigating Dec 15, 2025, 01:37 PM UTC
Ubicloud runners are currently experiencing a partial outage, resulting in high queue for job requests. We are investigating the incident.
- monitoring Dec 15, 2025, 02:11 PM UTC
We've mitigated the issue and are actively monitoring the results. Queue times have started to improve, but due to the high load caused by the incident, some delays are still expected. We’ll share another update once everything is fully resolved. Thank you for your patience.
- resolved Dec 15, 2025, 05:07 PM UTC
Our provisioning times are back to normal.
Ubicloud Website experiencing issues
Timeline · 3 updates
- identified Nov 18, 2025, 01:18 PM UTC
Ubicloud website uses Cloudflare WAF. The incident https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 is causing downtime on the web pages of Ubicloud.
- monitoring Nov 18, 2025, 02:44 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 18, 2025, 02:55 PM UTC
This incident has been resolved.