IONOS Cloud Outage History

**What happened** Virtual machines with dedicated CPU allocations in the Frankfurt data center began exhibiting abnormally high CPU steal time indicating that the hypervisor was unable to provide the requested CPU resources. This performance degradation occurred on multiple customer instances and persisted even after guest operating system reboots and configuration changes. This lead to performance degradation in customer workloads. **How was that possible? \(Root cause\)** The issue was caused by a regression introduced during recent improvements to the virtual machine checkpointing mechanism. A code change intended to optimize the checkpoint/restore process inadvertently affected the live migration code path, which is used when VMs are moved between physical servers. The virtualization hypervisor uses a process called CPU pinning to ensure that guest VM threads run on their dedicated CPU cores as allocated. When this pinning is not properly configured, the VM's processes fall back to inheriting the CPU assignment of the parent process. In this case, the parent process's assignment included core 0 - a core normally reserved exclusively for the host operating system that should not be allocated to guest workloads. The result was a failure of the resource allocation guarantees: * VMs with dedicated vCPU allocations could not be pinned to their assigned cores * The VM's virtual CPU threads instead competed for time on an oversubscribed, host-reserved core * The hypervisor scheduler could not guarantee the VM's promised CPU time High CPU steal time resulted despite adequate physical CPU resources being available. This issue affected VMs in the Frankfurt infrastructure that underwent live migration operations during the deployment window when the problematic code was active. **How we prevent recurrence** _Enhanced CPU Pinning Validation_: The virtualization infrastructure codebase has been updated to restore proper CPU pinning for all live migration operations.. \(DONE\) _Strengthened Pre-deployment Testing_: Enhance the validation procedures for virtualization infrastructure changes to catch CPU allocation anomalies before code is deployed to production. \(DONE\) _Automated Abnormal Steal Time Alertin_g: Implement automated monitoring and alerting to detect abnormal CPU steal time on VMs with dedicated vCPU allocations. This will enable faster detection of similar configuration regressions in the future. \(Within Q2 2026\) _Enhancing post rollout monitoring_: Extend the post-deployment monitoring and assessment window to increase likelihood of anomalies being spotted and correctly correlated to a change. \(DONE\) **Closing remark** The incident resulted in measurable performance degradation for customers over an extended period. The fact that the increase in steal time initially went unnoticed highlighted a gap in our alerting and monitoring setup. The ambiguity of the symptoms - characterized by general performance issues in some guests -, and apparent lack of “common denominators” in affected systems meant that initial incident reports and existing indicators were not properly understood, correlated. and attributed in a timely fashion. The corrective actions outlined above address both the immediate defect and the systemic factors that allowed it to surface. These measures are designed to prevent recurrence and significantly reduce the time to detection and resolution in the future. We thank our affected customers and partners for their patience and constructive collaboration throughout this incident.

**What Happened?** On March 14, 2026 a subset of customer Managed Kubernetes control planes experienced time periods of intermittent, recurring unavailability. The incident persisted until March 16, 2026, when the last anomalies were recorded and the incident was fully mitigated. During the impact window, affected clusters were periodically unreachable, meaning operations depending on the Kubernetes API - such as deployments, scaling operations, and health checks - would not have worked reliably. **How Was This Possible?** The root cause was a combination of three factors that compounded each other: * Excessive data volume from a small number of clusters: A subset of clusters were storing unusually large amounts of data in the shared control plane database - primarily security scanning reports and policy audit records, alongside high volumes of event and autoscaling objects. Three clusters alone accounted for approximately 64% of all stored data on one of the affected database instances, putting significant pressure on shared resources. * Database maintenance frequency: Our control plane databases were configured to compact and reclaim unused space, given the high rate of data being written by the affected clusters, this interval was insufficient to keep up, causing the database to accumulate excessive historical revisions and grow beyond normal operating size. * Database fragmentation: As the database grew, the actual allocated size exceeded the amount of data actively in use - adding roughly 40% overhead. This pushed the database toward its hard storage limit, at which point it would have become read-only and caused a full control plane failure for all clusters on that instance. Together, these factors caused periodic stalls during database maintenance cycles, which temporarily disrupted the Kubernetes API server and made affected control planes unavailable for several minutes each time. **What We Are Doing to Prevent Recurrence?** We have already taken immediate action and have a structured plan in place for further improvements:: * Performed defragmentation on the affected database instances, reclaiming approximately 3–4 GiB per instance and restoring healthy operating headroom \(DONE\) * Reduced the database compaction intervals on the affected instances \(DONE\) * Expanded monitoring dashboards to improve visibility into control plane health per database instance \(DONE\) * Migrating the highest-volume clusters to dedicated database backends, eliminating the noisy-neighbor risk for those workloads entirely \(DONE\) * Improving alerting to ensure incidents of this nature trigger immediate paging notifications, ensuring timely human intervention. \(DONE\) * Rolling out updated compaction interval across all shared control plane database instances \(DONE\) * Implementing automated, scheduled defragmentation jobs for all instances to prevent fragmentation from building up \(DONE\) * Rebalancing cluster distribution across database instances to reduce concentration risk \(DONE\) **Short-term \(within 4 weeks\):** Reaching out directly to customers whose clusters are generating disproportionately high data volumes to discuss workload optimization, such as using external storage backends for large security scan reports instead of storing them in the control plane database \(ONGOING\) **Medium-term \(1–3 months\):** Specification of quotas: Introducing per-tenant resource quotas and key-count limits on the shared control plane database to prevent any single cluster from impacting others **Long-term \(Q2 2026 and beyond\):** * Re-architecting the shared control plane to provide stronger isolation between customer workloads, including dedicated database instances and improved vertical scaling * Evaluating full database sharding for customers with consistently high data volumes * Conducting regular operational drills focused on database resource exhaustion and recovery to improve our response time for future incidents

**What happened?** A network control-plane failure in the TXL data center caused progressive service degradation and a partial outage for one cluster. The incident resulted in intermittent connectivity loss for a subset of customers, with traffic impact ranging from 5% to 20% during three distinct intervals on **11.03.2026**: * 05:20 – 05:25 UTC * 11:58 – 12:10 UT * 17:47 – 18:08 UTC The issue resolved automatically once the affected network devices were rebooted sequentially and additional configuration changes have been applied. The incidents prompted a series of emergency maintenance to stabilize the cluster. **How was this possible? \(Root Cause\)** The underlying cause was the exhaustion of multicast forwarding resources on the switches serving the affected topology. This trigger is technically identical to the previous incident that occurred on [03.03.2026](https://status.ionos.cloud/incidents/3pyhtpglqf43). When the forwarding table reaches its maximum capacity, the network control plane cannot program required updates into the forwarding tables of the switches. This continuous failure to push updates overwhelms the system, resulting in severe CPU overload and an out-of-memory \(OOM\) crash of the control-plane process. Without this process, the fabric is unable to maintain stable forwarding, ultimately leading to BGP session flaps and the observed loss of connectivity. **What are we doing to prevent recurrence?** As part of the measures established following the [03.03.2026 ](https://status.ionos.cloud/incidents/3pyhtpglqf43)incident, we had already rolled out configuration changes to one of the two topologies in the affected cluster. These measures aim to significantly reduce the load of the control-plane by optimizing resweep and failover times. Due to existing instabilities in the second topology, these changes had not yet been applied there. In response to the degradations observed on **11.03.2026**, we accelerated the rollout of these optimizations via Emergency Maintenance across all TXL topologies and several other data centers yesterday. Post-implementation, we observed a significant positive impact, with OpenSM loads being substantially reduced. Furthermore we’ve reconfigured management services to significantly reduce the multicast forwarding entries resulting in a significantly lower base load of the control-plane process. **Immediate Technical Actions:** * _Resource Management_: Adjusted control-plane configurations to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table. \(DONE\) * _Multicast Load Reduction:_ Implemented measures to decrease the number of multicast group memberships, further reducing pressure on the forwarding table. \(DONE\) * _Proactive Monitoring:_ Established continuous monitoring of forwarding table utilization and control-plane memory to trigger early alerts before capacity limits are reached. \(DONE\) **Long-term Structural Improvements:** * _Control-Plane Isolation:_ We are migrating our control-plane to dedicated, high-performance servers. This work, which began in Q4 2025, ensures the separation of network management from data-plane traffic to eliminate resource competition. We are also performing a deep-dive audit of IPv6 configurations and IPoIB driver settings. \(Ongoing, ETA: Q2 2026\) * _Network Modernization:_ Our interconnect fabric is undergoing a strategic modernization program—including switches, gateways, and drivers—to increase both resiliency and performance. \(Ongoing, ETA: Q3 2026\) With these measures now in place, we are confident the immediate cause of the cluster instability has been mitigated. Our long-term strategy will further enhance network stability and scalability across all data centers while addressing the identified bottlenecks. We recognize that these disruptions have impacted our customers and partners, and we sincerely appreciate your patience regarding the short-notice emergency maintenance announced yesterday. We are continuing to monitor the cluster closely to ensure all deployed fixes remain effective.

We want to share the Root Cause Analysis to this incident: **What happened** On 4 March 2026 customers of the IONOS Container Registry experienced 504 Gateway Timeout errors when pushing or pulling container images. Deployments that relied on the registry were blocked. **How was that possible \(Root Cause\)** The registry runs on IONOS Managed Kubernetes \(MK8s\) infrastructure. A temporary capacity constraint caused two critical control‑plane components to be placed on the same proxy instance instead of distributing them across separate proxies. This happened despite existing anti-affinity rules. The shared proxy reached its maximum concurrent‑connection limit and stopped accepting new connections. Because all registry traffic to the Kubernetes API traverses this proxy, push and pull operations failed with 504 errors. The migration created the co‑location condition; the connection‑limit exhaustion was the direct trigger. **What we are doing to prevent recurrence** Immediate \(completed\) * Provisioned additional proxy capacity. * Relocated the affected control‑plane components onto separate proxy instances, restoring balanced load and ending the 504 errors. Short‑term * Architectural redesign: Redesign registry‑to‑API connectivity so each node uses a dedicated local proxy, eliminating shared‑proxy bottlenecks. Design validated in test environments; production rollout scheduled for Q2 2026. * Alert‑threshold review: Adjust alerting thresholds to trigger warnings before proxy connection utilization approaches capacity. Rollout in progress, expected completion Q2 2026. Mid‑term * Load redistribution: Deploy additional infrastructure clusters and redistribute existing registries to ensure no single cluster exceeds safe operating capacity. Automation will continuously balance load as usage grows. **Closing remarks** The outage directly impacted container‑image delivery and delayed customer deployments. We have restored full service and implemented concrete architectural and operational changes to eliminate the identified bottleneck.

**What happened?** Users experienced the Data Center Designer \(DCD\) UI becoming unresponsive, displaying a persistent "In Progress" hourglass popup when attempting to open or edit certain Virtual Data Centers \(VDCs\). This issue prevented users from editing components or loading VDCs entirely. The problem was resolved via a rollback of the affected version and the subsequent deployment of a fixed release. **How was this possible? \(Root Cause\)** Changes introduced in a recent release - which targeted storage volume handling - inadvertently impacted CDROM volumes. While CDROM volumes are technically classified as storage volumes within the codebase, they carry different properties. The logic implemented for HDD/SSD volumes did not account for these differences, causing the UI to enter a broken state when CDROM volumes were encountered. **What are we doing to prevent recurrence?** * Immediate Fix: We have implemented dedicated handling for CDROM volume properties to correctly differentiate them from HDD/SSD volumes within the storage logic. The fixed release has been published. \(DONE\) * Improved Documentation: Documentation for CDROM entities has been updated to increase visibility and awareness for future development work involving storage volumes. \(DONE\) * Quality Assurance and Pre-Release Testing: Structural improvements are being made to ensure more extensive testing of various scenarios and user journeys. This will increase test coverage and allow for a higher detection rate of edge cases prior to release. \(Ongoing – Q2 2026\)

# **Root Cause Analysis** Today we are releasing a preliminary Root Cause Analysis that represents the current state of the investigation focusing on the network incident. While we have high confidence in the technical details concerning the network incident, we will add to this analysis as new information becomes available related to the services affected by the spillover effects of this main incident. We expect to publish the complete RCA by the end of this week. _UPDATE 13.03.2026 - We wanted to provide an update to the published preliminary RCA to enrich the findings and increase transparency related to other affected services._ ## **What happened?** During a [scheduled maintenance window](https://status.ionos.cloud/incidents/zg4mpk9x724t) to replace a faulty switch in a cluster situated in the TXL data center, a network outage occurred. The incident triggered a spillover effect, resulting in a significant queue of unprocessed provisioning jobs from the affected cluster. Following network recovery, this backlog generated significant resource locking delays, slowing processing of queued jobs. This impacted systems and services relying on self-healing automations triggered by the primary incident or undergoing changes at that time. _UPDATE: In the following we want to explain in more details the impact on other affected services:_ **IAM:** For customers, the DCD frontend was unavailable during the incident. This was due to the fact that the underlying IAM service was not reachable for the frontend. During the incident, an IP assignment maintenance job had been triggered, which was pending completion for an extended period of time, rendering the service unavailable to the frontend. **AI Model Hub:** The AI Model hub service was affected directly by the network outage. While the service itself remained functional throughout the incident, a network connectivity loss on the Managed Kubernetes Cluster meant that it was unavailable for customers. **DBaaS/Managed Kubernetes:** DBaaS and Managed Kubernetes were impacted through two separate mechanisms. First, servers hosting DBaaS and Kubernetes workloads experienced a direct loss of BGP connectivity as a result of the network outage, causing immediate service disruption. Second, the resulting delays in provisioning queue processing caused key operations such as volume attach/detach activities as well as automatically triggered self-healing mechanisms to be deferred, leading to further delays during the recovery phase when network connectivity was restored. **Provisioning:** Provisioning was first directly impacted by the connectivity loss to resources on the affected cluster. This led to an initial spike in the processing queue jobs. As services affected by the connectivity issues entered self-healing additional jobs were placed in the queue. This led to an exponential increase in job numbers, which, after the network incident was resolved, led to resource locking bottlenecks, which reduced the normal processing speed of the queue. This hampered automated recovering mechanisms, leading to extended service degradations, especially for IAM and Managed Kubernetes. ## **How was this possible? \(Technical Root Cause\)** The network maintenance took place in Topology 2 of 2 in the affected cluster. During the maintenance, Topology 2 of 2 was disabled as planned. At this point, the Multicast Forwarding Table \(MFT\) capacity across all switches in Topology 1 was already strained due to saturation from IPoIB multicast groups. Due to this, Topology 1 was already operating in a vulnerable state. Topology 1 of 2 then suffered a critical gateway overload, initiated by a link flap on a switch which flooded the InfiniBand control plane with alerts, triggering a fabric resweep - an automated self-healing mechanism aiming to reset the network map. Because the MFT was already at capacity, the reprogramming required for the reset failed. The resulting surge of unsuccessful route updates overwhelmed the management layer, causing BGP sessions to drop and ultimately severing connectivity for hosts in the affected cluster. As the connectivity loss made it impossible for provisioning jobs to succeed in the affected cluster all changes scheduled for resources on the affected hosts were queued. The network incident was resolved by a switchover from the affected Control Plane to its standby, a reboot of the locked-up gateways, and recovery of Topology 1 from the scheduled maintenance. Although the network incident was resolved, the resulting volume of provisioning jobs triggered significant resource locking. This bottleneck delayed critical updates, leaving several \(self-healing\) services in a degraded state as they waited for their queued jobs to process. ## **What are we doing to prevent recurrence?** Today, we are highlighting the measures - both planned and currently deployed - designed to reduce the likelihood of a similar incident. We are listing these measures by affected service, but want to underline that the “interconnectedness” of the services through the provisioning queue was identified as a key contributor to the duration of the service degradation. While individual measures will make the services individually more robust, we are also implementing architectural changes to the provisioning queue to ensure better de-coupling and reduce the risk of spillovers as observed here. ### Network In an RCA released for a related incident on [11.03.2026](https://manage.statuspage.io/pages/xdhr50sc5fkm/incidents/0rmsrc0dnk5h#postmortem) triggered by the same technical root cause, we have published immediate technical actions and long-term structural improvements. To improve consistency, we decided to share these measures and their implementation status here, as well. Immediate Technical Actions: * Resource Management: Adjusted control-plane configurations to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table. \(DONE\) * Multicast Load Reduction: Implemented measures to decrease the number of multicast group memberships, further reducing pressure on the forwarding table. \(DONE\) * Proactive Monitoring: Established continuous monitoring of forwarding table utilization and control-plane memory to trigger early alerts before capacity limits are reached. \(DONE\) Long-term Structural Improvements: * Control-Plane Isolation: We are migrating our control-plane to dedicated, high-performance servers. This work, which began in Q4 2025, ensures the separation of network management from data-plane traffic to eliminate resource competition. We are also performing a deep-dive audit of IPv6 configurations and IPoIB driver settings. \(Ongoing, ETA: Q2 2026\) * Network Modernization: Our interconnect fabric is undergoing a strategic modernization program - including switches, gateways, and drivers - to increase both resiliency and performance. \(Ongoing, ETA: Q3 2026\) With these measures now in place, we are confident the immediate cause of the cluster instability has been mitigated. Our long-term strategy will further enhance network stability and scalability across all data centers addressing the identified bottlenecks ### IAM Immediate Technical Actions: IAM Service Resilience during Maintenance: We have hardened IAM maintenance protocols to ensure critical identity services remain functional even if primary Cloud APIs are temporarily unavailable. \(DONE\) Long-term Structural Improvements: Hardening Core Identity Services \(IAM\): We are expediting the running migration of our Identity and Access Management \(IAM\) system to a dedicated platform. This removes IAM's dependency on the standard managed Kubernetes clusters \(Q3 2026\). ### AI Modelhub As the AI Modelhub was affected by connectivity issues caused by issues in the underlying Managed Kubernetes setup, the service will directly benefit from the measures planned to increase Managed Kubernetes resilience. ### DBaaS/Managed Kubernetes Improvements to Management and Provisioning Handling: Structural improvements are planned that make cluster setup and management more seamless, reliable, and automated. This will minimize the surface for service disruptions during updates and changes, like those triggered by self-healing mechanisms. Key improvements include better handling of node maintenance, more predictable scaling, and enhanced stability for production workloads. These changes are part of a broader effort to improve the Kubernetes experience, but will also help address the specific issues relevant in this incident by introducing a dedicated provisioning provider. \(Q3 2026\) ### Provisioning Enhanced Provisioning Resilience: To prevent "logjams" during maintenance or incidents, we are introducing automated circuit breakers within our provisioning engine to reduce the risk for overloads and perpetual queue buildup caused by self-healing services. Several quality of service improvements are planned to improve visibility and control over job execution and prioritization within the provisioning queue. Additional steps towards preparing a de-coupling of provisioning queues will be made in an upcoming provisioning maintenance to test provisioning switchover from one DC to another. \(Q1 2026\) We hope this update to our preliminary RCA increases transparency regarding how the network incident in the affected cluster unfolded and how it influenced other services. We have derived a series of improvements from this event that will help make our services more resilient in the future. While our ongoing network modernization initiative will provide greater performance and stability, we also aim to reduce the 'blast radius' of future incidents by identifying and addressing dependencies within our services. We recognize that this incident has affected customers and partners in various ways. It is important to us to provide a comprehensive, transparent account of the disruption, as well as the initiatives we have implemented - and will continue to put into place - to help avoid similar issues moving forward.

**Incident Summary** On March 2, 2026, some Managed Kubernetes clusters in our Frankfurt \(DE/FRA\) region became stuck in an `UPDATING` or `DEPLOYING` state. This caused intermittent communication issues between control plane components and worker nodes. **Root Cause** The instability was caused by our control plane management system running out of memory and crashing under high load. When the system restarted, it was unable to properly resume its interrupted tasks, which left several clusters stuck mid-update. **Resolution** Our engineering teams quickly stabilized the system by increasing its memory allocation. Once stable, engineers manually recovered the remaining clusters that were stuck to restore full service. **Prevention** To prevent this from happening again, we are working to address an upstream software bug related to how the system handles memory-related crashes. Additionally, we are fixing an internal alerting issue that failed to notify our on-call team, ensuring a much faster response should a similar issue arise in the future.

We want to share the following Root Cause Analysis with you _Update: 25.02.2026 - Initial publishing_ **What happened?** On February 24, 2026, during a scheduled maintenance window intended to improve network resiliency in the FRA region \(\[[Link](https://status.ionos.cloud/incidents/y3q7703x5fg4)\]\), a configuration deployment led to a loss of connectivity for public services. Internal monitoring detected the outage shortly after the start of the announced maintenance, once the configuration change was applied. Our network engineering teams identified link failures between critical hardware components. By 20:55 UTC, a manual fix was applied to the affected devices, and initially affected services were restored by 21:00 UTC. **How could this happen? \(Root Cause\)** The incident was caused by a configuration state drift between our central software repository and the live hardware settings in the FR7 production environment that went undetected before the rollout. Specifically, the outage involved Forward Error Correction \(FEC\) settings—parameters that allow different brands of networking hardware to communicate reliably. * **The Discrepancy:** Unlike the staging environments, the production environment had unique configurations for settings required for multi-vendor hardware interoperability. Despite using a "four-eyes" principle to validate the configuration changes, the rendered output did not provide enough visibility into the unexpected discrepancy. This caused the difference to go unnoticed. * **The Trigger:** The automated deployment performed a "full rebuild" of the device configuration. Because the repository did not contain the specific FEC settings \(the discrepancy\), it omitted them during the rebuild. * **The Result:** Once the new configuration was pushed, the lack of FEC parity caused the physical links between mismatched devices to fail, dropping traffic for all public services in the region. **What are we doing to prevent recurrence?** We are committed to ensuring this specific failure mode does not happen again. Our engineering teams have initiated the following corrective actions: * **Comprehensive Configuration Audit:** We are performing a full audit of all production devices to identify and resolve any "drifts" where live settings \(like FEC\) differ from our central repository. \(to be completed within Q1 2026\) * **Improved Validation Checks:** We are implementing an automated pre-flight check that compares the "intended" configuration against the "running" configuration to clearly flag any potential omissions before a change is finalized. This increases the visibility of drifts and unexpected discrepancies and reduces the surface area for human error. \(to be completed within Q1 2026\) The scheduled maintenance was initially planned to have only a few seconds of service disruption; however, due to the issues described, it caused a significantly higher impact. This maintenance was part of our ongoing initiative to improve stability and performance in our data centers. Our network team remains committed to driving this initiative forward. We understand the impact that this incident has caused and are working with due diligence and urgency to incorporate the lessons learned to further reduce risks during maintenance operations on our core network components.