Is Exoscale down?

We are investigating an increased error rates and latencies on the API. We’ll post an update as soon as we have more information.

We are investigating increased errors rate with Compute related API calls

We have reverted a compute orchestrator release that was being deployed. We are monitoring for situation improvements.

The situation did not improve. We are continuing investigations. All Mutating compute related API calls are currently impacted.

The root cause has been identified, we are working towards mitigation.

Mitigation have been applied. We are monitoring the situation for recovery

All systems are back to normal. We continue to monitor the situation

The issue has been resolved

We are investigating an increased error rates and latencies on the API. We’ll post an update as soon as we have more information.

The impact is confined to the ch-gva-2 zone, the API is working correctly in other zones.

The source of the problem has been localized and we are actively working on applying mitigations.

We are still actively working on mitigations, the impact is extended api, nlb, sks and ai services in ch-gva-2.

We are still actively working on applying mitigations, services are still unreachable.

We are still actively working on applying mitigations. NLB services are correctly processing traffic and were not impacted, we are removing them from the impacted scope.

The team is confirming that all known mitigations have been applied. Another part of the team is investigating packet drops on NLB in Geneva.

All mitigations have been applied and the team is monitoring the system. Investigations into the NLB packet loss is ongoing.

Mitigations are still been applied, they did not solve the ongoing incident. The impact continues on both API services (full impact) & NLB traffic (partial packet loss).

New mitigations are being applied, we will publish new updates shortly.

We confirmed some services are recovering, some flaps are to be expected in the coming minutes.

Our systems are back to normal, we are monitoring the situation. Every services should be back online.

The situation is stable, we are closing this status. A public postmortem will be published in the coming days.

Post-mortem Summary On 2026-10-08, between 13:18 and 14:52 UTC, the Exoscale public API in ch-gva-2, the global Portal, the Network Load Balancer (NLB) service in ch-gva-2 and the Scalable Kubernetes Service (SKS) in ch-gva-2 were unavailable or heavily degraded. During infrastructure work to prepare a new Exoscale zone, a configuration change was applied by mistake to our production platform in ch-gva-2. This change detached firewall rules (security groups) from the nodes that host our reverse proxies, NLBs and SKS control planes. Inbound traffic to these nodes was then dropped. Recovery took longer than we wanted. Several of the internal tools we normally use to fix this kind of problem relied upon the services that were down. We restored the missing rules through an emergency procedure, and the first attempt did not take effect, which delayed recovery by about one hour. Customer instances kept running for the whole incident. Data was not affected. Impact From 13:18 to 14:52 UTC: Public API (ch-gva-2): unavailable. Many clients (the exo CLI, Terraform and other integrations) use ch-gva-2 by default, for example to list zones. Because of this, creating, updating, deleting and listing resources failed or hung in all zones. Network Load Balancers (ch-gva-2): unavailable or intermittent. Services exposed through an NLB were unreachable or flapping. SKS (ch-gva-2): the Kubernetes control plane of clusters was unreachable. Worker nodes and running workloads were not affected unless they were exposed through an NLB. Portal (global): unavailable. From 14:52 to about 15:45 UTC, there were residual effects: Some SKS background operations failed because one of our internal secrets backend nodes were overwhelmed after service was restored. Some cross-zone bucket replication into ch-gva-2 timed out. All replication tasks recovered afterwards and got re-executed. Timeline UTC 13:18: A configuration change meant for a new zone is applied to the ch-gva-2 production platform by mistake. Security groups are detached from the reverse-proxy, NLB and SKS control-plane nodes. The public API, the Portal and NLBs become unreachable. 13:24: Incident declared, response team assembled. 13:40 -> 14:40: The team restores the security group associations through an emergency procedure. Our usual tooling cannot be used because it depends on the impacted API. The changes do not take effect on the hypervisors, and traffic is still dropped. 14:40: Root cause of the failed restoration found: the corrections had been written to a read replica of the database instead of the primary. 14:47: Corrections applied to the primary database. 14:48: NLB nodes start recovering. 14:50: Public API and exo CLI operational again. 14:52: Portal operational again. End of main impact. 15:06: SKS health checks green in all zones. 15:20 -> 15:40: An internal secrets backend node, overloaded by the backlog of requests, causes some SKS background jobs to fail. The node is restarted at 15:34 and the service is healthy again at 15:40. ~15:45: Bucket replication errors into ch-gva-2 stop. 15:57: Incident closed. Total duration of main impact: 1h34. Root cause Our infrastructure-as-code configuration is shared across zones. A change prepared for the new zone was planned and applied against the ch-gva-2 production platform instead of the new platform. It removed the security groups that allow inbound traffic on the nodes hosting our reverse proxies, NLBs and SKS control planes. Two factors made the incident last longer: Circular dependencies: The deployment pipeline that applied the change lost access to the API and could not complete or report its state. Several internal tools and communication channels were also hosted behind the impacted reverse proxies. Emergency procedure error: Because the API was down, we had to restore the security groups at a lower level. Using direct writes to the database opened us up to the risk of human errors, which caused us to initially write the changes incorrectly. As a result, they had no effect for about one hour. What We Learned Infrastructure changes need stronger safeguards. Our IAC configuration is shared across zones and makes it possible for a change to impact multiple zones. While we already have strong safeguards against any resource deletion, we need to improve our safeguards to catch removals in mutating changes. Emergency procedures must be reliable. Restoring network configuration without the public API was error-prone and slow. We need to improve our tooling for these situations, gaining efficiency and confidence while also avoiding direct database access. Global operations depend too much on one zone. Our public, internal tooling and integrations use ch-gva-2 by default, so a regional outage affected API operations in every zone. We are actively working on each of these points. We apologize for the disruption. We know many of you depend on the Exoscale API and NLBs for your own services, and we are committed to making sure this kind of change cannot have this impact again.

We are investigating an increased error rates and latencies on the API. We’ll post an update as soon as we have more information.

API connectivity is impacted for for most of the services. The issue is related to some internal connectivity issue we are facing. We are working to identify the root cause.

We identified the root cause and applied mitigation

We are monitoring the services recovery

Services are back to normal.

It appears we are still experiencing transient API issues in some zones. We are investigating

The service should be back to normal. We are monitoring the situation

The situation is still unstable. We are working back on mitigation

Mitigation have been applied. Services are back to normal. We are monitoring the situation

The issue has been resolved.

Customers with dedicated deployments might encounter HTTP 500 errors when making inference requests.

We discovered the issue and have already fixed some deployments. We are continuing to apply the mitigation.

All the deployments have been fixed, we are monitoring the situation.

The situation has been resolved; we will monitor it until tomorrow morning.

We haven’t seen any more HTTP 500 errors since our last communication. The situation is nominal; we are sorry for the inconvenience. A public post-mortem will follow.

Post-mortem Summary On October 1, 2026, during a scheduled maintenance window to migrate to a new AI router, a partial outage occurred affecting Dedicated Inference services. Customers initially received HTTP 404 errors, followed by HTTP 502 errors. The root cause was identified as missing TLS configurations in some customer deployments. Impact Requests to impacted Dedicated Inference deployments resulted in errors for approximately 1.5 hours. This impacted all zones where dedicated inference is available. Timeline 17:00 CET: Migration to the new vLLM router started. 17:15 CET: DNS change that resulted in routing issues, causing requests to end up with 404 errors. 17:30 CET: The routing issue has been fixed, and HTTP 502 errors were being returned 18:00 CET: The incident was escalated as a partial outage 18:07 CET: Mitigations are being gradually applied to fix the TLS configurations of the deployments 18:21 CET: All deployments across all zones have been successfully patched. 18:30 CET: Service back to nominal What Happened The new AI Router proxies to Dedicated inference deployments through mTLS. Some of the mTLS resources were missing for some deployments. Because of this misconfiguration, impacted deployments returned HTTP 502 errors for all inference requests. What We Learned We didn’t have a proper way to validate that all deployments had their mTLS configuration set up before the maintenance, and we incorrectly assumed this was true for all of them. What We’ve Changed We took an inventory and created the missing resources. Moving Forward Future infrastructural rollouts will be performed more gradually, progressing deployment by deployment, staged per zone. Also, we are adding more processes to ensure the invariants that should hold are true before performing this kind of maintenance.

We are experiencing issues with a scheduled NLB maintenance. API is not working, while no traffic drop to existing NLBs is expected. We are working on the fix. Sorry for the inconvenience.

The migration has being completed. No more issues are expected.