Resolved after 2 hours 30 minutes, affecting Global and Portal and 1 more, on October 8, 2026.
Post-mortem Summary On 2026-10-08, between 13:18 and 14:52 UTC, the Exoscale public API in ch-gva-2, the global Portal, the Network Load Balancer (NLB) service in ch-gva-2 and the Scalable Kubernetes Service (SKS) in ch-gva-2 were unavailable or heavily degraded. During infrastructure work to prepare a new Exoscale zone, a configuration change was applied by mistake to our production platform in ch-gva-2. This change detached firewall rules (security groups) from the nodes that host our reverse proxies, NLBs and SKS control planes. Inbound traffic to these nodes was then dropped. Recovery took longer than we wanted. Several of the internal tools we normally use to fix this kind of problem relied upon the services that were down. We restored the missing rules through an emergency procedure, and the first attempt did not take effect, which delayed recovery by about one hour. Customer instances kept running for the whole incident. Data was not affected. Impact From 13:18 to 14:52 UTC: Public API (ch-gva-2): unavailable. Many clients (the exo CLI, Terraform and other integrations) use ch-gva-2 by default, for example to list zones. Because of this, creating, updating, deleting and listing resources failed or hung in all zones. Network Load Balancers (ch-gva-2): unavailable or intermittent. Services exposed through an NLB were unreachable or flapping. SKS (ch-gva-2): the Kubernetes control plane of clusters was unreachable. Worker nodes and running workloads were not affected unless they were exposed through an NLB. Portal (global): unavailable. From 14:52 to about 15:45 UTC, there were residual effects: Some SKS background operations failed because one of our internal secrets backend nodes were overwhelmed after service was restored. Some cross-zone bucket replication into ch-gva-2 timed out. All replication tasks recovered afterwards and got re-executed. Timeline UTC 13:18: A configuration change meant for a new zone is applied to the ch-gva-2 production platform by mistake. Security groups are detached from the reverse-proxy, NLB and SKS control-plane nodes. The public API, the Portal and NLBs become unreachable. 13:24: Incident declared, response team assembled. 13:40 -> 14:40: The team restores the security group associations through an emergency procedure. Our usual tooling cannot be used because it depends on the impacted API. The changes do not take effect on the hypervisors, and traffic is still dropped. 14:40: Root cause of the failed restoration found: the corrections had been written to a read replica of the database instead of the primary. 14:47: Corrections applied to the primary database. 14:48: NLB nodes start recovering. 14:50: Public API and exo CLI operational again. 14:52: Portal operational again. End of main impact. 15:06: SKS health checks green in all zones. 15:20 -> 15:40: An internal secrets backend node, overloaded by the backlog of requests, causes some SKS background jobs to fail. The node is restarted at 15:34 and the service is healthy again at 15:40. ~15:45: Bucket replication errors into ch-gva-2 stop. 15:57: Incident closed. Total duration of main impact: 1h34. Root cause Our infrastructure-as-code configuration is shared across zones. A change prepared for the new zone was planned and applied against the ch-gva-2 production platform instead of the new platform. It removed the security groups that allow inbound traffic on the nodes hosting our reverse proxies, NLBs and SKS control planes. Two factors made the incident last longer: Circular dependencies: The deployment pipeline that applied the change lost access to the API and could not complete or report its state. Several internal tools and communication channels were also hosted behind the impacted reverse proxies. Emergency procedure error: Because the API was down, we had to restore the security groups at a lower level. Using direct writes to the database opened us up to the risk of human errors, which caused us to initially write the changes incorrectly. As a result, they had no effect for about one hour. What We Learned Infrastructure changes need stronger safeguards. Our IAC configuration is shared across zones and makes it possible for a change to impact multiple zones. While we already have strong safeguards against any resource deletion, we need to improve our safeguards to catch removals in mutating changes. Emergency procedures must be reliable. Restoring network configuration without the public API was error-prone and slow. We need to improve our tooling for these situations, gaining efficiency and confidence while also avoiding direct database access. Global operations depend too much on one zone. Our public, internal tooling and integrations use ch-gva-2 by default, so a regional outage affected API operations in every zone. We are actively working on each of these points. We apologize for the disruption. We know many of you depend on the Exoscale API and NLBs for your own services, and we are committed to making sure this kind of change cannot have this impact again.
Affected components
Update timeline
- investigating Oct 08, 2026, 01:27 PM UTC
We are investigating an increased error rates and latencies on the API. We’ll post an update as soon as we have more information.
- investigating Oct 08, 2026, 01:31 PM UTC
The impact is confined to the ch-gva-2 zone, the API is working correctly in other zones.
- investigating Oct 08, 2026, 01:37 PM UTC
The source of the problem has been localized and we are actively working on applying mitigations.
- investigating Oct 08, 2026, 01:44 PM UTC
We are still actively working on mitigations, the impact is extended api, nlb, sks and ai services in ch-gva-2.
- investigating Oct 08, 2026, 01:53 PM UTC
We are still actively working on applying mitigations, services are still unreachable.
- investigating Oct 08, 2026, 01:59 PM UTC
We are still actively working on applying mitigations. NLB services are correctly processing traffic and were not impacted, we are removing them from the impacted scope.
- investigating Oct 08, 2026, 02:05 PM UTC
The team is confirming that all known mitigations have been applied. Another part of the team is investigating packet drops on NLB in Geneva.
- investigating Oct 08, 2026, 02:22 PM UTC
All mitigations have been applied and the team is monitoring the system. Investigations into the NLB packet loss is ongoing.
- investigating Oct 08, 2026, 02:33 PM UTC
Mitigations are still been applied, they did not solve the ongoing incident. The impact continues on both API services (full impact) & NLB traffic (partial packet loss).
- investigating Oct 08, 2026, 02:44 PM UTC
New mitigations are being applied, we will publish new updates shortly.
- investigating Oct 08, 2026, 02:50 PM UTC
We confirmed some services are recovering, some flaps are to be expected in the coming minutes.
- monitoring Oct 08, 2026, 02:54 PM UTC
Our systems are back to normal, we are monitoring the situation. Every services should be back online.
- resolved Oct 08, 2026, 03:57 PM UTC
The situation is stable, we are closing this status. A public postmortem will be published in the coming days.
- investigating Oct 09, 2026, 11:48 AM UTC
Post-mortem Summary On 2026-10-08, between 13:18 and 14:52 UTC, the Exoscale public API in ch-gva-2, the global Portal, the Network Load Balancer (NLB) service in ch-gva-2 and the Scalable Kubernetes Service (SKS) in ch-gva-2 were unavailable or heavily degraded. During infrastructure work to prepare a new Exoscale zone, a configuration change was applied by mistake to our production platform in ch-gva-2. This change detached firewall rules (security groups) from the nodes that host our reverse proxies, NLBs and SKS control planes. Inbound traffic to these nodes was then dropped. Recovery took longer than we wanted. Several of the internal tools we normally use to fix this kind of problem relied upon the services that were down. We restored the missing rules through an emergency procedure, and the first attempt did not take effect, which delayed recovery by about one hour. Customer instances kept running for the whole incident. Data was not affected. Impact From 13:18 to 14:52 UTC: Public API (ch-gva-2): unavailable. Many clients (the exo CLI, Terraform and other integrations) use ch-gva-2 by default, for example to list zones. Because of this, creating, updating, deleting and listing resources failed or hung in all zones. Network Load Balancers (ch-gva-2): unavailable or intermittent. Services exposed through an NLB were unreachable or flapping. SKS (ch-gva-2): the Kubernetes control plane of clusters was unreachable. Worker nodes and running workloads were not affected unless they were exposed through an NLB. Portal (global): unavailable. From 14:52 to about 15:45 UTC, there were residual effects: Some SKS background operations failed because one of our internal secrets backend nodes were overwhelmed after service was restored. Some cross-zone bucket replication into ch-gva-2 timed out. All replication tasks recovered afterwards and got re-executed. Timeline UTC 13:18: A configuration change meant for a new zone is applied to the ch-gva-2 production platform by mistake. Security groups are detached from the reverse-proxy, NLB and SKS control-plane nodes. The public API, the Portal and NLBs become unreachable. 13:24: Incident declared, response team assembled. 13:40 -> 14:40: The team restores the security group associations through an emergency procedure. Our usual tooling cannot be used because it depends on the impacted API. The changes do not take effect on the hypervisors, and traffic is still dropped. 14:40: Root cause of the failed restoration found: the corrections had been written to a read replica of the database instead of the primary. 14:47: Corrections applied to the primary database. 14:48: NLB nodes start recovering. 14:50: Public API and exo CLI operational again. 14:52: Portal operational again. End of main impact. 15:06: SKS health checks green in all zones. 15:20 -> 15:40: An internal secrets backend node, overloaded by the backlog of requests, causes some SKS background jobs to fail. The node is restarted at 15:34 and the service is healthy again at 15:40. ~15:45: Bucket replication errors into ch-gva-2 stop. 15:57: Incident closed. Total duration of main impact: 1h34. Root cause Our infrastructure-as-code configuration is shared across zones. A change prepared for the new zone was planned and applied against the ch-gva-2 production platform instead of the new platform. It removed the security groups that allow inbound traffic on the nodes hosting our reverse proxies, NLBs and SKS control planes. Two factors made the incident last longer: Circular dependencies: The deployment pipeline that applied the change lost access to the API and could not complete or report its state. Several internal tools and communication channels were also hosted behind the impacted reverse proxies. Emergency procedure error: Because the API was down, we had to restore the security groups at a lower level. Using direct writes to the database opened us up to the risk of human errors, which caused us to initially write the changes incorrectly. As a result, they had no effect for about one hour. What We Learned Infrastructure changes need stronger safeguards. Our IAC configuration is shared across zones and makes it possible for a change to impact multiple zones. While we already have strong safeguards against any resource deletion, we need to improve our safeguards to catch removals in mutating changes. Emergency procedures must be reliable. Restoring network configuration without the public API was error-prone and slow. We need to improve our tooling for these situations, gaining efficiency and confidence while also avoiding direct database access. Global operations depend too much on one zone. Our public, internal tooling and integrations use ch-gva-2 by default, so a regional outage affected API operations in every zone. We are actively working on each of these points. We apologize for the disruption. We know many of you depend on the Exoscale API and NLBs for your own services, and we are committed to making sure this kind of change cannot have this impact again.
Your product runs on more than Exoscale
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services