Is Medusa down?

We are currently investigating an issue with new deployments not succeeding. Existing production deployments are not affected.

We are continuing to investigate this issue.

Issue has been resolved and new deployments are going through; we are now looking into the culprit.

Previous fix resolved the issue and deployments have fully recovered. We are now working on strengthening this part of our infrastructure.

We are currently investigating this issue. Customer applications are not affected.

We have identified the issue and are rolling out a fix now.

This incident has been resolved. The issue was with a faulty migration. No data was impacted.

We are currently investigating this issue.

We are continuing to investigate this issue.

We have identified the issue and are working on resolving it.

A fix has been pushed and we are seeing services recovering.

This incident has been resolved and service have recovered. We will be sharing more information once we have properly analyzed the issue.

## Summary On September 16, 2026, Medusa Cloud experienced a partial outage affecting applications running in our EU region during an upgrade of our networking infrastructure. The upgrade introduced a version of Knative Serving/Kourier with a startup performance regression. In our EU data plane, which manages a large number of application routes, the affected controllers could not finish loading routing state before their health-check deadline. Kubernetes therefore restarted them repeatedly. Because the controllers were unable to fully initialize the routing layer after the upgrade, some application traffic could not be routed successfully and returned 502 or 503 responses. The primary customer impact lasted approximately **25 minutes, from 13:30 to 13:55 UTC**. ## Impact The incident affected applications hosted in the **EU region**. The outage was partial rather than complete. Some requests continued to succeed throughout the incident, while a significant portion of requests returned 502 or 503 responses. Applications in other Medusa Cloud regions were not affected. ## Timeline All times are UTC. **13:27** — We began upgrading Knative Serving and Kourier in the EU data plane. **13:28** — New versions of the routing controllers began starting. **13:29** — The new controllers began failing their health checks before completing initialization and were restarted by Kubernetes. **13:30** — Customer-facing errors increased sharply, with requests beginning to return 502 and 503 responses. **13:31–13:54** — The affected controllers repeatedly attempted to start, failed to complete initialization within their health-check window, and restarted. We investigated the controller startup behavior and worked to restore the routing layer. **13:54** — We increased the resources available to the affected controllers, allowing them to complete initialization. **~13:55** — Customer applications began recovering and traffic returned to normal. **16:12** — We upgraded Knative to version 1.22, which contains the upstream fix for the startup regression. ## Root cause The incident was caused by a startup performance regression in the version of Knative/Kourier deployed during the upgrade. During startup, Kourier's controller reconstructs the routing state for applications in the cluster. A regression in the affected version caused this process to perform inefficiently when initializing a large number of routes. In our EU data plane, initialization consequently took longer than the controller's configured health-check window. Kubernetes interpreted the delayed startup as an unhealthy controller and restarted it. Because all newly upgraded controller replicas encountered the same condition, they entered a restart loop and could not reliably provide the routing state required by the Kourier gateways. The regression was subsequently fixed upstream and is included in Knative 1.22. For technical reference, the upstream issue and fix are documented in the [net-kourier issue](https://github.com/knative-extensions/net-kourier/issues/1423) and [corresponding fix](https://github.com/knative-extensions/net-kourier/pull/1441). ## Resolution We first restored service by increasing the resources available to the Kourier controllers, which allowed them to complete initialization and repopulate the required routing state. After service had recovered, we upgraded Knative to version 1.22, which contains the upstream fix for the regression. ## Preventive actions We are taking the following actions to reduce the likelihood and impact of similar incidents: * **Pin critical networking components to validated versions** before rolling them into production. * **Strengthen upgrade validation** for infrastructure components, including startup behavior at the scale of our production environments. * **Improve monitoring of the routing controllers** so degraded startup behavior can be detected earlier. * **Review the resilience of our ingress architecture** to reduce the impact of controller initialization failures during future upgrades. We apologize for the disruption this incident caused to customers running workloads in our EU region.

We've identified an issue with new deployments not completing successfully. A fix is currently being rolled out, and we are monitoring for resolution.

The issue has been resolved and deployments are completing successfully again. Production workloads were not impacted, as this only affected new deployments. The culprit to the issue was a missing database migration containing a queried column in deployment pipeline.