DataFlair Outage History

We have detected a very high I/O delay on xeon01. At the moment, we suspect that the issue may be related to unusually high resource consumption caused by individual customer VMs. We are currently analyzing the situation in detail to identify the exact root cause and take appropriate action. Further updates will follow as soon as more information becomes available.

We are currently experiencing issues with the IPv4 subnet 77.90.52.0/24 at the legacy location. Our team is actively investigating the root cause and working on a resolution. To ensure a fast mitigation, we are offering affected customers a free IP address change. Please open a support ticket so we can proceed with the reallocation immediately.

A few hours after closing the previous incident, continuous monitoring and customer feedback indicated that the read/write performance of the RAID subsystem on epyc01 (FC cluster) is below expected levels. We are currently conducting a more detailed analysis to identify and permanently resolve the underlying cause of the I/O latency. Further updates will be provided as soon as new information becomes available.

We are currently investigating I/O-related performance issues on node epyc01. Our engineering team is analyzing disk and storage performance metrics to identify the root cause. Further updates will be provided as soon as more information becomes available.

We are currently investigating a global outage affecting the host ryzen7950x3d-2. This system is impacted by the ongoing migration from the legacy cluster to the new cluster. As part of this process, the affected host will be replaced as soon as possible. Further information and timelines will be shared once available.

For the past few hours, our website has been inaccessible from certain providers. We are actively investigating the root cause of this issue and are working on a solution to restore full connectivity as quickly as possible. Further updates will be provided as soon as more information becomes available.

So far, we have not identified any anomalies within our infrastructure. Several third-party providers are already involved and are reviewing the situation on their end as well. At this time, we have found no indication of malicious traffic or similar activity. We are continuing to perform a comprehensive review across all relevant components. Further updates will follow as new information becomes available.

Unfortunately, we have been experiencing issues with a VDS system for several hours. As a result, the affected system needs to be replaced on short notice. We are currently evaluating the available options and will provide an update here as soon as possible.

We apologize for the lack of transparency in this incident. Our team is currently making every effort to restore the affected system. Unfortunately, it appears that we are dealing with a severe failure, which is preventing the system from returning to a stable operational state at this time. Further updates will be provided as soon as more information becomes available.

As of 15:19 CET today, the node ryzen393 is offline. We are already in active coordination with a partner to determine the root cause of the outage. Further updates will be provided here as soon as more information becomes available.

Our internal monitoring system has detected a significant performance degradation on the nodes epyc7543-3 and epyc7543-4. We are actively investigating the root cause of this issue to restore normal operation as quickly as possible. At this time, we are experiencing an unprecedented surge in support tickets related to this matter. Due to this high volume, we may not be able to respond individually to every ticket promptly. We kindly ask that you subscribe to the newsletter thread for this incident to receive continuous updates, rather than opening new tickets. Thank you for your understanding and patience as we work diligently to resolve this.

The Ryzen 9 9950X node restarted itself. We’re looking into this and suspect a hardware issue. We’re emptying the affected node and investigating it further to avoid any downtime.

We are beginning the migration of Ryzen and high-end root servers to firstcolo FRA4. Starting today at 20:00 CET, voluntary customers can already be migrated to the new location. Please note that the server’s IPv4 address will change, but in the meantime, you will receive an upgraded CPU — AMD Ryzen 9 9950(X3D). If you wish to migrate, please open a support ticket in the customer panel with the subject: "Server Migration – Status" Requests will be processed gradually, and there will be no additional costs. The full migration of all nodes will begin on Monday, 10.11.2025, at which point migration will be mandatory.

Almost all customers have now migrated from the high-end root servers (based on EPYC Genoa). We are migrating the last few and will then provide new resources for the next migrations.

The migration of the remaining servers will begin today.

Today we are beginning the migration of the remaining Ryzen VDS servers, specifically: ryzen7950x3d-2 ryzen7950x3d-3 Some of these servers will be upgraded to the new AMD Ryzen 9 9950X (3D) CPUs, while others will continue running on the AMD Ryzen 9 7950X (3D), depending on available space. Please note that during this migration, the IPv4 address of each server will change. For customers with multiple servers, the number of IPv4 addresses you have booked will remain the same after migration—you will receive an equivalent number of new IPs. Once the migration is complete, affected customers will receive a brief notification email. The new IP addresses will then be available in the customer panel.

Due to technical issues with the Ryzen03 node, we had to temporarily pause the migration process. The problem has now been fully resolved, and the migration will continue tonight from 10:00 PM. Customers who would like to be migrated in advance from ryzen7950x3d-2 or ryzen7950x3d-3 are welcome to contact us via our ticket support. Please note that, as part of the migration, both your IPv4 and IPv6 addresses will be changed. Thank you very much for your understanding and patience.

English: We will restart the node at 12 PM to identify and fix an error. German: Wir werden den Node um 12 Uhr mittags neu starten, um einen Fehler zu identifizieren und zu beheben.

Yesterday's problem had not yet been resolved. We are currently checking everything.

We have to restart the system once to rule out further sources of error.

ryzen7950x3d-4 a node of our AMD Ryzen 9 7050x3d cluster is currently throwing an error. We are identifying this.

For a few days now, epyc9334-2 has been experiencing repeated outages, and we are currently checking this.

A critical security vulnerability has been identified in Proxmox. As a precaution, we are performing urgent upgrades on all affected host systems to ensure infrastructure security and stability. Affected Hosts: epyc7443-1 epyc7543-1, epyc7543-2, epyc7543-3 epyc9334-1, epyc9334-2 epyc9754-2 ryzzen7950x3d-1, ryzzen7950x3d-2, ryzzen7950x3d-3, ryzzen7950x3d-4 Please note: These upgrades may result in temporary downtimes. We are working to complete the process as quickly and smoothly as possible. Thank you for your understanding.

The following nodes must be restarted immediately: epyc7443-1, epyc7543-1, epyc7543-2, epyc9334-1, epyc9334-2, epyc9754-2.