Experiencing degraded availability in EU central region

Update timeline

1. investigating Sep 16, 2026, 01:49 PM UTC

   We are currently investigating this issue.

2. investigating Sep 16, 2026, 01:51 PM UTC

   We are continuing to investigate this issue.

3. identified Sep 16, 2026, 01:51 PM UTC

   We have identified the issue and are working on resolving it.

4. monitoring Sep 16, 2026, 01:56 PM UTC

   A fix has been pushed and we are seeing services recovering.

5. resolved Sep 16, 2026, 02:01 PM UTC

   This incident has been resolved and service have recovered. We will be sharing more information once we have properly analyzed the issue.

6. postmortem Sep 21, 2026, 06:12 AM UTC

   ## Summary On September 16, 2026, Medusa Cloud experienced a partial outage affecting applications running in our EU region during an upgrade of our networking infrastructure. The upgrade introduced a version of Knative Serving/Kourier with a startup performance regression. In our EU data plane, which manages a large number of application routes, the affected controllers could not finish loading routing state before their health-check deadline. Kubernetes therefore restarted them repeatedly. Because the controllers were unable to fully initialize the routing layer after the upgrade, some application traffic could not be routed successfully and returned 502 or 503 responses. The primary customer impact lasted approximately **25 minutes, from 13:30 to 13:55 UTC**. ## Impact The incident affected applications hosted in the **EU region**. The outage was partial rather than complete. Some requests continued to succeed throughout the incident, while a significant portion of requests returned 502 or 503 responses. Applications in other Medusa Cloud regions were not affected. ## Timeline All times are UTC. **13:27** — We began upgrading Knative Serving and Kourier in the EU data plane. **13:28** — New versions of the routing controllers began starting. **13:29** — The new controllers began failing their health checks before completing initialization and were restarted by Kubernetes. **13:30** — Customer-facing errors increased sharply, with requests beginning to return 502 and 503 responses. **13:31–13:54** — The affected controllers repeatedly attempted to start, failed to complete initialization within their health-check window, and restarted. We investigated the controller startup behavior and worked to restore the routing layer. **13:54** — We increased the resources available to the affected controllers, allowing them to complete initialization. **~13:55** — Customer applications began recovering and traffic returned to normal. **16:12** — We upgraded Knative to version 1.22, which contains the upstream fix for the startup regression. ## Root cause The incident was caused by a startup performance regression in the version of Knative/Kourier deployed during the upgrade. During startup, Kourier's controller reconstructs the routing state for applications in the cluster. A regression in the affected version caused this process to perform inefficiently when initializing a large number of routes. In our EU data plane, initialization consequently took longer than the controller's configured health-check window. Kubernetes interpreted the delayed startup as an unhealthy controller and restarted it. Because all newly upgraded controller replicas encountered the same condition, they entered a restart loop and could not reliably provide the routing state required by the Kourier gateways. The regression was subsequently fixed upstream and is included in Knative 1.22. For technical reference, the upstream issue and fix are documented in the [net-kourier issue](https://github.com/knative-extensions/net-kourier/issues/1423) and [corresponding fix](https://github.com/knative-extensions/net-kourier/pull/1441). ## Resolution We first restored service by increasing the resources available to the Kourier controllers, which allowed them to complete initialization and repopulate the required routing state. After service had recovered, we upgraded Knative to version 1.22, which contains the upstream fix for the regression. ## Preventive actions We are taking the following actions to reduce the likelihood and impact of similar incidents: * **Pin critical networking components to validated versions** before rolling them into production. * **Strengthen upgrade validation** for infrastructure components, including startup behavior at the scale of our production environments. * **Improve monitoring of the routing controllers** so degraded startup behavior can be detected earlier. * **Review the resilience of our ingress architecture** to reduce the impact of controller initialization failures during future upgrades. We apologize for the disruption this incident caused to customers running workloads in our EU region.