IONOS Cloud Outage History

Started Jul 15, 2026, 10:15 AM UTC· resolved Jul 17, 2026, 04:55 PM UTC

1. identified Jul 15, 2026, 10:15 AM UTC

   While phone support will generally be available, our capacity is reduced due to multiple cases of sick leave. We want to inform you that our phone support will be limited during the following time slots, leading to increased response times on the phone channel. 15.07.2026: 21:00 – 05:00 UTC 16.07.2026: 21:00 – 05:00 UTC If possible, we ask that you submit tickets using the DCD Form or via email. Thank you for your understanding!

2. resolved Jul 17, 2026, 04:55 PM UTC

   Phone coverage is currently normal

Read the full incident report →

Started Jul 13, 2026, 11:18 AM UTC· resolved Jul 13, 2026, 11:21 AM UTC

1. investigating Jul 13, 2026, 11:18 AM UTC

   We are currently investigating problems accessing VMs and DBaaS service located in Frankfurt. We will inform you as soon as the functionality has been restored.

2. resolved Jul 13, 2026, 11:21 AM UTC

   This was a duplicate.

Read the full incident report →

Started Jul 13, 2026, 11:16 AM UTC· resolved Jul 13, 2026, 01:44 PM UTC

1. investigating Jul 13, 2026, 11:16 AM UTC

   We are currently experiencing an issue where resources in the FRA region are unreachable and showing an Unknown state. This issue is currently being investigated and we will keep you informed of any updates.

2. investigating Jul 13, 2026, 11:17 AM UTC

   We are continuing to investigate this issue.

3. identified Jul 13, 2026, 11:51 AM UTC

   Our teams identified some affected hardware servers that caused the unavailability. We are now in process of migrating the affected VMs to different HW servers, so please check your VMs. We are still working on the root cause investigation.

4. monitoring Jul 13, 2026, 12:00 PM UTC

   Our tech teams were able to restore all VMs and products, so all are back available and running. However, we keep monitoring the situation for a while.

5. resolved Jul 13, 2026, 01:44 PM UTC

   The incident has been resolved, the VMs are back working again.

Read the full incident report →

Started Jul 09, 2026, 10:16 PM UTC· resolved Jul 10, 2026, 07:59 AM UTC

1. investigating Jul 09, 2026, 10:16 PM UTC

   We are currently responding to an incident in our VIT datacenter. We will regularly update this status page with information.

2. identified Jul 09, 2026, 10:29 PM UTC

   Power has been restored and site support systems are back in operational state. We have started recovering infrastructure components to fully restore service.

3. identified Jul 09, 2026, 10:47 PM UTC

   Most affected hosts have been brought back into service. We are working on restoring the remaining components.

4. identified Jul 10, 2026, 12:28 AM UTC

   All but one hosts are recovered. The remaining host suffered a hardware defect and needs to be replaced to recover the workload provisioned there. This is currently underway. We estimate that this work will be completed within the next minutes.

5. monitoring Jul 10, 2026, 12:41 AM UTC

   The last host and VMs have been recovered. We are setting this incident into monitoring status. Preliminary information on the Incident: Following a power grid outage, the site suffered a partial interruption of power, which impacted several datacenter support systems. The reason for this is under active investigation. This affected the availability of a subset of infrastructure components and hosts, subsequently impacting the VMs provisioned on them or dependent on those components. A full Root Cause Analysis (RCA) is currently being compiled and will be shared here once completed. Thank you for your patience.

6. resolved Jul 10, 2026, 07:59 AM UTC

   We are marking this incident as resolved. The Root Cause Analysis will be published here once compiled.

7. postmortem Jul 22, 2026, 09:55 AM UTC

   **What happened?** On July 9, 2026, the Logroño datacenter experienced a cascading power and cooling failure impacting customer workloads and services. The incident began at 21:59 CEST and required on-site manual intervention to restore service. Full power was restored at 23:05 CEST; complete service normalization concluded at 09:22 CEST on July 10. No customer data loss has been identified. **How was this possible? \(Root Cause\)** The outage was caused by a single electrical fault that triggered a cascading sequence of automated safety responses. At 21:59 CEST, a residual current device \(RCD breaker D1\) in the high-voltage control cabinet tripped. At this point, no visible outage occurred. However, the trip silently disabled the generator control system - the generators lost visibility of their circuit breakers and were no longer able to start automatically. The facility appeared fully operational, but its backup power capability was gone. 23 minutes later, at 22:22 CEST, the 66 kV high-voltage grid supply failed. The protection system detected overcurrent and initiated an automatic safety shutdown - a designed safety response, not a malfunction. Because the generators had been silently disabled since 21:59, they were unable to take over the load. All UPS systems switched to battery mode, protecting the workload throughout. However, as cooling infrastructure is not UPS-backed, all chillers and CRAH units lost power immediately. Manual intervention by on-site staff restored grid power at 22:50 CEST. At 23:05 CEST, all generators were reset and the transformer breaker was closed, fully restoring power to the facility. The sequential service recovery process then began. The exact cause of the initial D1 trip is still under investigation by our electrical engineering team and an external vendor performing forensic analysis on-site. **What we are doing to prevent recurrence** The investigation has already identified the architectural root cause: a single RCD breaker acting as a shared upstream dependency for both the generator control system and the high-voltage protection system. The following measures are being implemented: Already completed * Generator validation: All diesel generators have been confirmed synchronized and operationally ready in auto-mode following the corrective actions applied on-site. * Infrastructure validation: On July 13, 2026, a scheduled maintenance shutdown by grid operator Iberdrola provided a real-world validation. All systems performed as designed — generators started automatically, no service interruption occurred. * High-voltage station vendor inspection completed — no additional defects found. Short-term - within 4 weeks * Redundant AC supply for HV protection and generator control system — eliminating the identified single point of failure so that no single component can simultaneously disable both primary and backup power paths. * Review of RCD sizing and placement in the HV auxiliary circuit. * Network failover review: engineering is reviewing route reflector and load balancer failover behavior to prevent static connectivity conditions that compounded recovery time. Structural * Redundant AC supply for HV protection and generator control system — eliminating the identified single point of failure so that no single component can simultaneously disable both primary and backup power paths. * Emergency runbook update: HV protection system failure scenario added and distributed to all on-site and on-call staff. * Full electrical architecture audit: a comprehensive review of the Logroño facility's electrical distribution layout to identify and remediate any further single points of failure. * Full post-incident review \(PIR\) to be completed and shared upon request. Closing remarks Despite the resilience measures already in place, service disruption could not be completely avoided during this event. The cascading nature of the failure - where a single circuit trip silently disabled backup power capability before the grid failure occurred - is a serious architectural finding, and we have moved immediately to eliminate that dependency. On July 13, 2026, our infrastructure was put to a real-world test during a scheduled Iberdrola maintenance shutdown. All systems performed as designed. We are confident that the corrective actions taken are effective. We recognize the impact this had on customers relying on our services and are committed to closing all identified action items with urgency and transparency.

Read the full incident report →

Started Jul 06, 2026, 10:14 AM UTC· resolved Jul 08, 2026, 09:31 AM UTC

1. identified Jul 06, 2026, 10:14 AM UTC

   We are currently investigating instances of performance degradation in job processing. Our Provisioning Team is currently working on fixing the root cause of these events.

2. identified Jul 07, 2026, 09:36 AM UTC

   Still, the global provisioning system is not working properly. Therefore, all other dependencies, like K8s and managed products, are affected, too. We know this is affecting customer setups, and that urgent changes are not processed properly, and we apologize for the inconvenience. We identified possible distractors, but fixing this is a complex task, with need to involve and coordinate several technical teams. We are working with highest prio to get the provisioning back working, and we will let you know asap on any progress.

3. monitoring Jul 08, 2026, 06:57 AM UTC

   Our Provisioning Team has been closely tracking queue performance over the last few hours and has noted steady improvements. We are moving this incident to Monitoring status. We will continue to monitor queue processing closely and will update this page on any changes.

4. resolved Jul 08, 2026, 09:31 AM UTC

   This incident has been resolved. We are working on the RCA and will publish it here asap

Read the full incident report →

Started Jul 02, 2026, 08:30 AM UTC· resolved Jul 03, 2026, 09:16 AM UTC

1. investigating Jul 02, 2026, 08:30 AM UTC

   Currently we are facing increased provisioning problems resulting in error code [VDC-5-0]. We understand the impact, and our teams are investigating the root cause to get this fixed asap. Please postpone your provisioning until we resolve the issue, we will inform you of any updates.

2. identified Jul 02, 2026, 01:51 PM UTC

   We found the root cause and implemented mitigation measures. We expect the errors to decrease significantly now. We will let you know asap when we consider the situation being back to normal.

3. resolved Jul 03, 2026, 09:16 AM UTC

   Our Provisioning went back operational yesterday around 16:30 UTC. Further monitoring showed no issues anymore, so this incident was resolved.

Read the full incident report →

Started Jul 02, 2026, 07:00 AM UTC· resolved Jul 02, 2026, 01:50 PM UTC

1. investigating Jul 02, 2026, 07:00 AM UTC

   IONOS Cloud Support is temporarily unavailable via phone. We ask Customers and Partners to contact Cloud Support via the DCD Form or Email, instead.

2. investigating Jul 02, 2026, 07:01 AM UTC

   We are continuing to investigate this issue.

3. resolved Jul 02, 2026, 01:50 PM UTC

   Phone Support is now available again

Read the full incident report →

Started Jun 30, 2026, 09:08 PM UTC· resolved Jul 07, 2026, 10:09 AM UTC

1. investigating Jun 30, 2026, 09:08 PM UTC

   Customers may experience connection problems to the control plane and degraded functionality of kubernetes

2. investigating Jun 30, 2026, 10:55 PM UTC

   The situation appears to have stabilized for the time being. An internal investigation into the exact root cause is ongoing.

3. monitoring Jul 01, 2026, 07:07 PM UTC

   Incident involving degradation of the control plane has been resolved and we are monitoring the situation.

4. resolved Jul 07, 2026, 10:09 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 23, 2026, 10:24 AM UTC· resolved Jun 23, 2026, 02:26 PM UTC

1. investigating Jun 23, 2026, 10:24 AM UTC

   Due to an internal issue that is currently being investigated, the BGE-M3 model is unavailable.

2. investigating Jun 23, 2026, 10:26 AM UTC

   We are continuing to investigate this issue.

3. monitoring Jun 23, 2026, 11:33 AM UTC

   A fix has been implemented and we are monitoring the service.

4. resolved Jun 23, 2026, 02:26 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 22, 2026, 07:07 PM UTC· resolved Aug 25, 2026, 05:24 PM UTC

1. investigating Jun 22, 2026, 07:07 PM UTC

   Due to exceptionally high demand in two clusters in our TXL and FRA datacenters, we have temporarily paused Managed Kubernetes Automated Maintenance. To ensure that clusters remain operational without interruption during updates, Automated Maintenance requires temporary cluster "cloning", which increases resource consumption. As most maintenance windows are scheduled to run over the weekend, resource demand peaks sharply during these periods. To protect cluster stability and prevent service degradation, we have paused maintenance for the affected clusters. Any maintenance scheduled from today onward will not run until we resume. We know automated Kubernetes maintenance is an important feature of our platform, and we did not take this decision lightly. We are expanding capacity in the affected clusters as a high priority and will update this page as soon as maintenance resumes.

2. identified Jun 23, 2026, 08:21 AM UTC

   We were able to reactivate the MK8s maintenances in all clusters in our TXL datacenter

3. resolved Aug 25, 2026, 05:24 PM UTC

   We have enabled maintenance again in all DCs.

Read the full incident report →

Started Jun 22, 2026, 08:01 AM UTC· resolved Jun 22, 2026, 11:04 AM UTC

1. investigating Jun 22, 2026, 08:01 AM UTC

   We are currently investigating issues with the MK8s control plane

2. monitoring Jun 22, 2026, 08:15 AM UTC

   Control Planes are reachable again. Our MK8s Team is currently investigating the root cause for the service interruption.

3. identified Jun 22, 2026, 08:35 AM UTC

   We have observed another spike and are setting the incident back to active. Our MK8s Team is informed and are actively working the incident.

4. resolved Jun 22, 2026, 11:04 AM UTC

   We have marked this incident as resolved. Several underlying VMs entered an unhealthy state, which increased the CPU load on the remaining healthy VMs supporting the service. This cascaded into the intermittent availability issues. All systems are now operating normally.

Read the full incident report →

Started Jun 19, 2026, 06:35 AM UTC· resolved Jun 22, 2026, 06:42 PM UTC

1. investigating Jun 19, 2026, 06:35 AM UTC

   We are currently investigating increased error rate for S3 in our FRA DC

2. identified Jun 19, 2026, 06:57 AM UTC

   We are reducing the impact of the incident, as our Object Storage Team has traced back the increased error rates to mainly SlowDown codes, which affect only heavily loaded buckets.

3. monitoring Jun 19, 2026, 09:03 AM UTC

   Error rate has dropped. We are monitoring the situation further.

4. resolved Jun 22, 2026, 06:42 PM UTC

   We are marking this incident as resolved, as no further abnormalities occurred.

Read the full incident report →

Started Jun 18, 2026, 03:04 PM UTC· resolved Sep 21, 2026, 08:31 AM UTC

1. identified Jun 18, 2026, 03:04 PM UTC

   We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one. What You Can Do - Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available. - Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success. - If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server. Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible. We sincerely appreciate your patience and understanding.

2. resolved Sep 21, 2026, 08:31 AM UTC

   Additional GPU capacity has been successfully provisioned, and standard availability is now restored. We are removing this note.

Read the full incident report →

Started Jun 17, 2026, 11:40 AM UTC· resolved Jun 17, 2026, 02:03 PM UTC

1. investigating Jun 17, 2026, 11:40 AM UTC

   We are currently experiencing a high error rate with S3 Object Storage in TXL - eu-central-3. This is currently being investigated and we will continue to keep this page updated.

2. resolved Jun 17, 2026, 02:03 PM UTC

   The incident has now been resolved, and accessing S3 Buckets should now be working as expected.

Read the full incident report →

Started Jun 15, 2026, 09:03 AM UTC· resolved Jun 15, 2026, 11:20 AM UTC

1. investigating Jun 15, 2026, 09:03 AM UTC

   We are currently investigating increased loading times and timeouts for the DCD.

2. identified Jun 15, 2026, 09:23 AM UTC

   Our DCD Team is currently investigating the incident. We have confirmed that Cloud API remains functional and available. Customers and Partners needing to deploy changes to their setup can still use ionosctl and cloudapi to do so.

3. identified Jun 15, 2026, 10:18 AM UTC

   We believe that the trigger for the incident has been mitigated. We are currently working on restoring all services related to the DCD.

4. monitoring Jun 15, 2026, 10:41 AM UTC

   The DCD is available again. We are monitoring the system closely and will set this incident to resolved once the environment is confirmed stable.

5. resolved Jun 15, 2026, 11:20 AM UTC

   The system is stable. We are resolving this incident. The trigger of this incident will be shared once the analysis has been completed.

Read the full incident report →

Started Jun 12, 2026, 07:44 AM UTC· resolved Jun 12, 2026, 01:49 PM UTC

1. investigating Jun 12, 2026, 07:44 AM UTC

   We are currently investigating increased error rate for S3 in eu-central-3

2. identified Jun 12, 2026, 07:54 AM UTC

   Our Object Storage Team has identified an issue connected with gateways in the affected region and is currently mitigating the problem. Customers may see connectivity issues and timeouts. We will keep you updated on the mitigation efforts.

3. identified Jun 12, 2026, 08:07 AM UTC

   The mitigation efforts have lead to first improvements. We have downgraded the incident severity. However, we still see elevated error rates and are working towards resolving the remaining issues.

4. monitoring Jun 12, 2026, 09:08 AM UTC

   Error rates have significantly decreased. We are monitoring the situation further and will resolve the incident if the situation stays as is.

5. resolved Jun 12, 2026, 01:49 PM UTC

   We are closing the incident as no further error spikes were recorded. We will share the trigger of this incident once the analysis has been conducted.

Read the full incident report →

Started Jun 08, 2026, 01:21 PM UTC· resolved Jun 08, 2026, 02:33 PM UTC

1. investigating Jun 08, 2026, 01:21 PM UTC

   Currently we face major availability problems in our Object Storage service in Frankfurt. You might encounter degraded access to your buckets and objects in de (eu-central-1). The team is investigating the issue and trying to fix it as soon as possible.

2. monitoring Jun 08, 2026, 01:33 PM UTC

   The team found the issue in a bad network setting / routing. This was fixed, so the service is back again. It happened 12:45 - 13:20 UTC today, but now all endpoints are available. We are monitoring the situation, and we will provide an analysis once the investigation was finished.

3. resolved Jun 08, 2026, 02:33 PM UTC

   The Object Storage endpoints are stable now.

Read the full incident report →

Started May 29, 2026, 08:23 PM UTC· resolved Jun 01, 2026, 05:06 PM UTC

1. investigating May 29, 2026, 08:23 PM UTC

   We are currently investigating increased latency and error count in the AI Model Hub

2. identified May 29, 2026, 08:43 PM UTC

   We are currently experiencing significant load, which is affecting the response times of our Modelhub. Our team is working on removing performance bottlenecks.

3. monitoring May 29, 2026, 10:31 PM UTC

   The Mistral Small 24B Instruct model is currently experiencing a high load. For users experiencing timeouts, we recommend switching to an alternative model, such as GPT-OSS-120B. The slow response times previously observed in Collections are no longer occurring.

4. resolved Jun 01, 2026, 05:06 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 28, 2026, 10:43 AM UTC· resolved May 28, 2026, 05:21 PM UTC

1. investigating May 28, 2026, 10:43 AM UTC

   We are currently investigating a connectivity issue related to VPN Gateways

2. identified May 28, 2026, 10:47 AM UTC

   Our VPN Team has identified the issue and is currently mitigating. Affected gateways are already recovering. We are monitoring recovery progress and will update this page.

3. monitoring May 28, 2026, 11:34 AM UTC

   all affected Gateways have recovered. We are monitoring the behavior and will mark the incident as resolved.

4. resolved May 28, 2026, 05:21 PM UTC

   We are marking this incident as resolved. Our team is currently investigating a link to a previous maintenance task on the VPN gateways. We will update the incident with the investigation results once they are available.

Read the full incident report →

Started May 20, 2026, 12:19 PM UTC· resolved May 20, 2026, 09:38 PM UTC

1. investigating May 20, 2026, 12:19 PM UTC

   We are currently facing a network issue in Worcester. Several customer NICs are down due to the outage, so connectivity is lost for the affected instances. Our team is currently working on re-establishing connectivity as soon as possible. We will keep you informed on any new information. Affected services: Network Location: Location GB/BHX

2. identified May 20, 2026, 01:42 PM UTC

   The team identified the root cause and is working on a fix now. We need to migrate several VMs to get them back online. We will keep you informed on any change.

3. monitoring May 20, 2026, 02:41 PM UTC

   The team was able to resolve the issue and bring all instances back online, so the network and connectivity is back now. We keep monitoring the situation for stability.

4. resolved May 20, 2026, 09:38 PM UTC

   We consider this incident as resolved. An RCA will follow as soon as it's compiled.

5. postmortem Jun 15, 2026, 12:53 PM UTC

   In this RCA we want to share our analysis of a network connectivity incident that affected virtual servers on 20 May 2026 in our Worcester datacenter. During this period, LANs associated with a subset of your virtual machines became unavailable due to a network configuration issue in the underlying cluster. We have documented the sequence of events, the technical root cause, and the measures we are implementing to prevent recurrence. **What happened?** On 20 May 2026 at approximately 11:19 UTC, LAN connectivity was lost for virtual machines hosted on a subset of physical servers in an affected cluster. The impact manifested as a complete loss of LAN connectivity for the virtual machines running on those servers. Engineering was alerted and began investigation. The root cause was identified, and the affected virtual machines were migrated to alternative physical servers with compatible interface speeds. LAN connectivity was fully restored by 14:27 UTC. **How was that possible? \(Root Cause\)** The affected cluster contained a mix of physical servers operating at different interface speeds: a subset of servers were running at 100Gbps, while the remainder of the cluster had been upgraded to 200Gbps. This mixed-speed configuration was present prior to the incident. Multicast groups - which underpin LAN connectivity between virtual machines - form at the interface speed of the highest-speed server participating at the time the group is created. By platform design, a server may join an existing group only if its interface speed is equal to or greater than the group's speed. A faster server can therefore join a slower group, and its presence does not raise the group's speed, but a slower server cannot join a faster group. Before the incident, the affected groups had been created at 100Gbps. The 200Gbps servers were able to participate in them because their speed exceeded the group speed, and their presence did not change it. As a result, LANs spanning both 100Gbps and 200Gbps hosts operated normally. On 20 May, a platform-level liveboot rollout of a gateway component caused multicast groups to be recreated across the cluster. During recreation, each group's speed was re-derived from the highest-speed server then participating. Any group spanning both 100Gbps and 200Gbps hosts therefore reformed at 200Gbps. The 100Gbps servers were now slower than the group and were rejected on rejoin. All multicast join requests from the affected 100Gbps servers failed, and the LANs hosted on those servers went down. The gateway liveboot was the operational trigger that caused multicast groups to be rebuilt. The underlying condition that made those rebuilds produce a connectivity failure was the presence of servers with 100Gbps interface speeds in an otherwise 200Gbps cluster. This disparity had not been detected prior to the event. **What we are doing to prevent recurrence** Completed * The affected virtual machines were migrated to physical servers operating at 200Gbps, restoring LAN connectivity. The 100Gbps servers have been removed from active customer workloads. In progress * Interface upgrades: The affected physical servers are being upgraded to 200Gbps network interface cards, bringing their interface speed in line with the rest of the cluster. This eliminates the mixed-speed condition entirely. \(ETA: Q3 2026\) * Cluster speed monitoring: We are implementing automated monitoring to detect physical servers operating at non-standard interface speeds within a cluster. This will ensure that any future speed disparity is identified and flagged before it can affect customer workloads during operational events such as gateway liveboots. \(ETA: Q3 2026\) Structural review We are reviewing our gateway liveboot and rollout procedures to assess whether multicast group recreation can be performed in a way that is resilient to mixed-speed cluster configurations, adding a safeguard independent of monitoring coverage. **Closing remarks** The loss of LAN connectivity, even for a bounded period, disrupts operational continuity, and we recognize the impact this incident had on your environment. The root condition - a speed disparity between servers in the same cluster - should have been identified before it could be exposed by a routine operational event. We have addressed the immediate impact, and the NIC upgrades and monitoring improvements underway will close the gap that allowed this configuration to persist undetected.

Read the full incident report →

Started May 15, 2026, 12:18 PM UTC· resolved May 19, 2026, 04:24 PM UTC

1. identified May 15, 2026, 12:18 PM UTC

   We are currently investigating service degradation for Control Planes of some Kubernetes clusters in the Frankfurt region (de/fra). This could impact K8s workload changes for several customers. Our teams are working on a fix.

2. monitoring May 15, 2026, 01:35 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 19, 2026, 04:24 PM UTC

   We are marking this incident as resolved. The root cause of the service degradation affecting Kubernetes Control Planes in FRA has been identified as a DDoS attack targeting customers hosted on our infrastructure. This incident has been documented here: https://status.ionos.cloud/incidents/kknrz6284jl0 The identified bottleneck led to the observed intermittent reachability issues and increased latency between Kubernetes etcd nodes for the affected customers.

Read the full incident report →

Started May 15, 2026, 11:59 AM UTC· resolved May 19, 2026, 04:17 PM UTC

1. investigating May 15, 2026, 11:59 AM UTC

   We are writing to inform you that we have been experiencing sporadic connection issues and substantial delays on packet delivery. Network technicians have started working on the occurrence immediately after detection and will isolate the problem and solve the issue as quick as possible. However, it is possible that there will be a certain degradation in connection quality affecting individual virtual resources. We will inform you as soon as the functionality has been restored.

2. monitoring May 15, 2026, 01:18 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 19, 2026, 04:17 PM UTC

   We are marking this incident as resolved. The Root Cause of the incident was a DDOS attack on customers hosted in our infrastructure. While the attack was largely successfully mitigated by our countermeasures a intermittent saturation of an affected link resulted in an increase of package loss. Our network team has evaluated the countermeasures' effectiveness and the identified bottleneck and is currently working on technical and organizational improvements to further decrease detection time for comparable events.

Read the full incident report →

Started May 13, 2026, 03:33 PM UTC· resolved May 18, 2026, 07:36 AM UTC

1. identified May 13, 2026, 03:33 PM UTC

   As documented in the updates to the recent Storage Incident in our Karlsruhe DC, several Virtual Data Centers (VDCs) are currently restricted from issuing new provisioning jobs while redundancy is being restored for affected VMs. Original Incident: https://status.ionos.cloud/incidents/p6pjqxzgkh1g With this announcement we want to increase awareness about this current limitation. Impact Scope: Only customers and VDCs impacted by the original storage incident are affected by these provisioning limitations. Symptoms: Users may encounter error code VDC-5-12 when attempting to create or modify resources. Existing Workloads: All currently running VMs remain operational We will close this announcement once redundancy, as well as provisioning functionality, is fully restored.

2. resolved May 18, 2026, 07:36 AM UTC

   Redundancy has been fully restored and no limitations related to provisioning should remain.

Read the full incident report →

Started May 12, 2026, 08:50 AM UTC· resolved May 13, 2026, 06:56 AM UTC

1. investigating May 12, 2026, 08:50 AM UTC

   We are aware that, starting this morning, multiple customers have received an automated email from our Accounting Team advising that their Umsatzsteuer- / VAT ID numbers could not be verified. We are currently investigating this issue, and expect to send out follow-up emails to the affected customers once we know more.

2. identified May 12, 2026, 02:38 PM UTC

   We can confirm that these emails were sent out in error. There are no customer 'to-do's / calls-to-action, and we will send out an informational update to all affected customers as soon as possible.

3. resolved May 13, 2026, 06:56 AM UTC

   This incident has been resolved, and all affected customers have been contacted directly via email.

Read the full incident report →

Started May 12, 2026, 06:15 AM UTC· resolved May 18, 2026, 07:38 AM UTC

1. investigating May 12, 2026, 06:15 AM UTC

   We are currently experiencing restrictions regarding our Storage service in one cluster in our Karlsruhe region. We are working to restore regularly operation of our services as quickly as possible.

2. identified May 12, 2026, 08:02 AM UTC

   Our developers restored the storage availability, we are now restarting the affected VMs to bring them back online.

3. monitoring May 12, 2026, 08:59 AM UTC

   All VMs and Storages and VDCs are back online and available. We keep monitoring the situation.

4. investigating May 12, 2026, 11:04 AM UTC

   Unfortunately, we ran into further issues with our Storages in Karlsruhe, so the team keeps on repairing. Affected VMs (and VDCs) are not available and have to be restarted once they get back online. We will keep you updated asap on any new information.

5. identified May 12, 2026, 11:23 AM UTC

   Our storage team is currently working on a firmware restore and upgrade on the affected storage servers and will then attempt to bring them back into service.

6. identified May 12, 2026, 12:49 PM UTC

   We found relevant hardware to be defective, this has to be replaced first in the data center. After the physical connection is back, we need to sync and update the storage servers to get the VMs back online. We are working as fast as we can to get this fixed, however, this might take some hours.

7. identified May 12, 2026, 01:53 PM UTC

   The spares have been made available. They are currently brought online to allow recovery to continue. We estimate that we will have another update on the progress within the hour.

8. identified May 12, 2026, 04:01 PM UTC

   We have made progress regarding the recovery, and are currently in the process of restarting affected Virtual Machines.

9. identified May 12, 2026, 07:23 PM UTC

   The team has encountered a blocker while restarting the remaining VMs due to connectivity issues between the replaced hosts and the storage servers. This has significantly increased the complexity of the recovery effort. We are currently collaborating with a specialized development team to bring the remaining systems back online. We will provide the next update at 20:00 UTC or before if there are meaningful updates.

10. identified May 12, 2026, 08:00 PM UTC

    Our development team is working on re-establishing connectivity sessions of the affected VMs so that the recovery effort can continue. The first runs are currently starting.

11. identified May 12, 2026, 08:12 PM UTC

    The team has succeeded in restoring the first VMs during the initial run. We are currently working on rolling out the restoration process further and will update on the progress.

12. identified May 12, 2026, 09:08 PM UTC

    We have successfully recovered additional VMs. However, because full storage redundancy has not yet been reached for these systems, Storage Provisioning Jobs will currently fail. We ask affected customers to refrain from making configuration changes or placing excessive load on these servers until redundancy is fully restored.

13. monitoring May 12, 2026, 11:06 PM UTC

    Our team has recovered the remaining VMs. It’s important to note that storage redundancy is not yet fully restored. Due to necessary data synchronization processes and the volume of data involved, full recovery will take several additional hours. Important notice for affected customers: - Load: To ensure stability of the recovered VMs, please try to avoid excessive storage load where possible. - Provisioning: Provisioning jobs against affected VMs will not be possible until redundancy is fully restored. Consequently, making state changes will not be possible during this time. - VM States: During the incident, the state of affected VMs may have changed; some may currently be in "shutoff" state. - Manual Intervention: To avoid unintended impact on customers for whom "shutoff" is the desired state, we cannot automatically restart all VMs without approval. Due to these constraints, we kindly ask affected customers to carefully check their resources and request state changes via support ticket ([email protected]) where necessary. This is only needed until redundancy is restored. We are moving this incident to Monitoring status while work to restore full redundancy continues. We will provide an update here once that work is complete. A detailed Root Cause Analysis (RCA) will be published subsequently, explaining the cause of the incident, the measures taken to restore service, and the steps implemented to prevent recurrence. We thank our customers and partners for their patience and cooperation in ensuring the full recovery of affected services and workloads.

14. resolved May 18, 2026, 07:38 AM UTC

    As redundancy has been fully restored, we are resolving this incident. A RCA is being prepared and we estimate that it will be shared here today or tomorrow.

15. postmortem May 20, 2026, 09:05 PM UTC

    **Root Cause Analysis** Between 11 and 12 May 2026, a cluster in our FKB datacenter was affected by two consecutive storage hardware failures, which led to service degradation and I/O outage for affected workloads spanning over 15 hours and limiting provisioning functionality in affected VDCs. In the following Root Cause Analysis, we explain the incident, identify the technical triggers of the outage, describe the work done during mitigation, and outline the measures we are taking to prevent a recurrence. **Technical Context** Because the failure mode was a cascading sequence of two hardware events on a redundant storage pair, we want to provide the technical context necessary to understand how a redundant system could be affected to this degree: To provide a resilient service, storage servers in the affected clusters are deployed in redundant pairs \(Leg A and Leg B\) that hold synchronized data. Each leg itself has built-in redundancy and fault tolerance - for example, via RAID configurations for disks. The two legs are designed to mitigate risk from hardware failures or issues that would render the resilience mechanisms of a single leg ineffective. Failure of one leg is largely transparent to the VM host, which - for customers - continues to operate normally. In a "Zero Leg" event - where connectivity to both legs is lost - VMs served by the affected storage experience an outage. This is the event that caused this incident. **What Happened** On 11 May 2026 , the first of two servers \(Leg A\) in a redundant storage backend pair began reporting multiple disk errors in its RAID array. The error pattern - UDMA CRC errors across several disks - was consistent with degradation of a shared physical interconnect \(SAS backplane or cabling\), rather than wear on any single disk. From this point onward, all dependent storage volumes were operating without their secondary failover path, served exclusively from Leg B - a single-leg event. As a forced array reassembly was initiated additional disks faulted during the rebuild in the afternoon, degrading the array below the recoverable threshold. A subsequent reboot brought the array up fully inactive, and Leg A was considered unrecoverable. The decision was made to begin migrating redundancy onto alternative storage systems, with the goal of restoring two-leg operation. Due to the volume of data that needed to be synchronized, this process was expected to take several hours. On 12 May at 05:13 UTC - while synchronization was still in progress - Leg B experienced an independent disk disconnection in its own RAID array. With Leg A already non-functional, this second failure eliminated all remaining redundancy in the pair. All dependent volumes dropped to zero available copies simultaneously, causing an I/O outage for the affected virtual machines. Leg B was rebooted at 05:38 UTC and its array force-assembled with the remaining healthy disks, which allowed a first wave of volume recovery to complete by 08:25 UTC. Additional disk failures on the same server then materialized at 10:59 UTC, triggering a second outage. At this point, the decision was taken to perform a complete server replacement: at approximately 14:30 UTC, all disks were physically transferred to new server hardware and the RAID arrays were rebuilt on the replacement system. The hardware swap introduced an additional recovery complication: the replacement server now carried a different storage network identity. Compute nodes across the cluster still held cached connection sessions pointing to the original hardware identity, and VM restart attempts subsequently failed. When a cleanup of stale connection mappings across affected compute nodes in the fleet was performed existing automation tooling proved ineffective. This significantly hampered recovery efforts. Leg B was confirmed fully operational at 20:45 UTC, and affected VMs began recovering. VMs where automatic recovery was not possible were remediated manually in the following hours. During the subsequent synchronization process to re-establish redundancy between the new legs, the provisioning service was deactivated in VDCs that contained VMs served by the affected storage pair. This meant customers were temporarily unable to roll out changes to affected VMs. Storage redundancy was fully restored on 15 May, once all affected volumes had been migrated back to a fully redundant configuration. At this point, the service was fully recovered. **How was that possible?\(Root Cause\)** The incident was caused by two independent hardware failures on the two servers of a single redundant storage pair, occurring several hours apart. Each failure on its own would not have caused customer impact. The combination of both, within the window between the first failure and the completion of redundancy migration, eliminated the redundancy that the architecture is designed to provide. _Primary failure - Leg A._ The errors observed on Leg A were consistent with degradation of a shared physical interconnect - a SAS backplane or cabling issue - rather than end-of-life wear on individual disks. This pattern explains both why multiple disks failed in a correlated way, and why repair attempts could not outpace degradation: the rebuild operations themselves exercise the interconnect, and a second disk faulted under rebuild load before the array could complete. A stale array member prevented full reassembly, and the server was effectively out of service after its reboot. _Secondary failure - Leg B._ While migration of redundancy to alternative storage was in progress, Leg B suffered an independent disk disconnection. The exact root cause of this second failure is still under investigation, but the symptoms were consistent with previously observed wear-related failures within the same hardware generation and make. _Recovery complication - network identity mismatch._ When the disks from Leg B were transferred into a replacement chassis, the new server presented a different storage network identity. Compute nodes in the cluster, however, retained cached connection sessions pointing to the original hardware identity. VM restart attempts therefore failed against the replacement hardware until those sessions were invalidated across every affected compute node. As available automation tooling proved ineffective specialized tooling needed to be developed and tested during the incident. In summary: a single-leg operation window - opened by the first hardware failure - coincided with an independent second hardware failure on the surviving leg before redundancy could be restored. The recovery path then required full hardware replacement, which surfaced inadequate automation in the storage-to-compute identity handover which prolonged the recovery process significantly. **What we are doing to prevent recurrence** We have grouped the remediation into completed tasks, immediate measures, and a mid-term reviews to address the gaps that turned a localized hardware failure into a prolonged customer-impacting outage. _Already taken:_ * **Complete server replacement:** All disks from the affected servers were physically transferred to verified replacement hardware, with RAID arrays fully rebuilt and verified on the new platform. * **Disk replacement and array verification:** All involved disks were replaced; arrays were rebuilt and integrity-verified before being returned to service. _Immediate \(June 2026\):_ * **Automated session migration on hardware replacement:** Update and validate tooling that automatically refreshes storage network identity mappings across all compute nodes when a storage server of this configuration is replaced in the manner required during this incident. This directly addresses the cleanup step that extended recovery by several hours. * **Single-leg exposure review:** Tighten the operational window during which a degraded redundant pair is allowed to operate on a single leg in configurations in locations where Dynamic Leg Swapping is not available. Dynamic Leg Swapping refers to a progress where a single leg failure initiates an automated sync to alternate storage. Repair on the affected leg can continue and depending on which redundancy recovery strategy is quicker \(repair vs migration\) the leg that is available quickest will be chosen. This bounds the time spent on repair efforts, simplifies decision-making during incident response, and shrinks the window in which a subsequent hardware failure can directly impact connected workloads. _Mid-term \(within Q3\):_ * **Dynamic Leg Swapping Rollout:** We assess the technical prerequisites to retrofit locations to support Dynamic Leg Swapping or provide alternative strategies to shorten \(automated\) redundancy restoration in data centers. * **Recovery Tool Assessment**:: The incident has shown that existing recovery tooling needs to be reviewed and validated to work against all existing hardware and configuration combinations in the fleet. * **Hardware generation review:** Comprehensive review of the affected storage hardware generation, covering firmware versions, backplane integrity, and environmental conditions. Failure patterns will be evaluated to rule out problematic combinations. Components showing early signs of degradation will be proactively replaced or upgraded. **Closing Remarks** We recognize that this incident produced a full I/O outage for the virtual machines in environments that depended on this storage pair, lasting over 15 hours, and that volumes operated without their secondary failover path for several hours before and after the incident. For any production workload, an outage and loss of redundancy of this duration is significant and unacceptable. This incident not only resulted in service disruption and subsequent degradation, but also led to significant effort for partners and customers working on their side to mitigate the resulting outage. We believe that the measures we have completed and have planned will effectively reduce the risk of a similar failure pattern. The planned review will help to identify and reduce risk factors further. The assessment of recovery tooling will help us recover quicker and increase the robustness of our incident response. Thank you for your patience during the outage and recovery and your engineering teams' constructive coordination and support throughout this event.

Read the full incident report →