DataFlair Outage History

Started Mar 22, 2025, 04:00 AM UTC· resolved Mar 22, 2025, 04:00 AM UTC

1. resolved Mar 22, 2025, 07:31 AM UTC

   89.144.35.0/24 and 89.144.25.0/24 were disabled today and the problem should now be resolved.

Read the full incident report →

Started Feb 27, 2025, 03:00 PM UTC· resolved Feb 27, 2025, 03:00 PM UTC

1. resolved Feb 27, 2025, 05:50 PM UTC

   We have detected a problem on ryzen7950x3d-3 which was not foreseeable. The system simply hung up - we are working on finding the error.

Read the full incident report →

Started Feb 26, 2025, 07:07 PM UTC· resolved Feb 26, 2025, 07:09 PM UTC

1. identified Feb 26, 2025, 07:07 PM UTC

   The host system epyc9334-2 has just performed a restart, which was not planned.

2. monitoring Feb 26, 2025, 07:08 PM UTC

   All VMs restart one after the other.

3. resolved Feb 26, 2025, 07:09 PM UTC

   All VMs are running again and should run without any problems. This closes the incident.

Read the full incident report →

Started Feb 22, 2025, 09:41 PM UTC· resolved Feb 26, 2025, 01:29 PM UTC

1. investigating Feb 22, 2025, 09:41 PM UTC

   Unfortunately, the VDS cluster is affected by a failure due to the virtualisation method. We are already working on finding a solution.

2. identified Feb 22, 2025, 09:52 PM UTC

   The synchronisation of the cluster has stopped due to Proxmox. We are working on a solution.

3. monitoring Feb 22, 2025, 09:52 PM UTC

   We have found a temporary solution to the problem. However, a further step will be necessary to fully rebuild the cluster. All services are fully accessible.

4. resolved Feb 26, 2025, 01:29 PM UTC

   We will leave the emergency solution in place for the time being. This has no risk and no performance restrictions.

Read the full incident report →

Started Feb 11, 2025, 01:00 PM UTC· resolved Feb 11, 2025, 07:55 PM UTC

1. identified Feb 11, 2025, 01:00 PM UTC

   There is a severe performance overload, which is why we have to restart the system to restore stability.

2. resolved Feb 11, 2025, 07:55 PM UTC

   The host system has now been running smoothly for several hours. We are therefore closing the incident.

Read the full incident report →

Started Feb 01, 2025, 01:59 PM UTC· resolved Feb 02, 2025, 02:39 AM UTC

1. investigating Feb 01, 2025, 01:59 PM UTC

   For about two days now there has been packet loss on the epyc9754-1 system. We are already doing everything in our power to fix the problem as soon as possible.

2. resolved Feb 02, 2025, 02:39 AM UTC

   Problem resolved

Read the full incident report →

Started Jan 28, 2025, 08:14 PM UTC· resolved Jan 28, 2025, 08:14 PM UTC

1. resolved Jan 28, 2025, 08:14 PM UTC

   Due to the high load on epyc9754-1, we had to reboot the system out of necessity. New hard disks were installed, but unfortunately they were not recognized by the system.

Read the full incident report →

Started Jan 23, 2025, 12:14 PM UTC· resolved Jan 23, 2025, 06:10 PM UTC

1. investigating Jan 23, 2025, 12:14 PM UTC

   There is an error in the VLAN at epyc296, we are working on identifying the error.

2. identified Jan 23, 2025, 12:14 PM UTC

   We suspect a fault in the internal network due to the move to the new rack. We are in the process of rectifying the fault.

3. resolved Jan 23, 2025, 06:10 PM UTC

   Solved Problem

4. postmortem Jan 23, 2025, 06:10 PM UTC

   # English: On January 23, 2025, at 09:00 CET, a server relocation was carried out within the maincubes FRA01 data center, during which the EPYC cluster was to be moved to a new server rack. During the relocation, the internal network was disrupted, causing the cluster's firewall to crash and block all traffic. The outage was identified at 13:14 CET, and the error analysis revealed an issue with the VLAN configuration. To resolve the incident, the VLAN of the cluster was thoroughly scanned to identify and eliminate potential error sources. After no further issues were found, the firewall was successfully restored through a reset. The issue was fully resolved at 17:25 CET, and the EPYC cluster has been operational since that time. The EPYC cluster has been running stably since the resolution of the issue at 17:25 CET, and the incident has been successfully closed. ‌ # German: Am 23. Januar 2025 wurde um 09:00 Uhr CET ein Server-Umzug innerhalb des Rechenzentrums maincubes FRA01 durchgeführt, bei dem der EPYC-Cluster in ein neues Server-Rack verlagert werden sollte. Während des Umzugs wurde das interne Netzwerk beeinträchtigt, was dazu führte, dass sich die Firewall des Clusters aufhängte und keinen Traffic mehr durchließ. Der Ausfall wurde um 13:14 CET erkannt, und bei der Fehleranalyse stellte sich heraus, dass ein Problem in der VLAN-Konfiguration vorlag. Um den Vorfall zu beheben, wurde zunächst das VLAN des Clusters vollständig gescannt, um mögliche Fehlerquellen zu identifizieren und auszuschließen. Nachdem keine weiteren Probleme festgestellt wurden, konnte die Firewall durch einen Reset erfolgreich wieder in Betrieb genommen werden. Das Problem wurde schließlich um 17:25 CET vollständig behoben, und der EPYC-Cluster war ab diesem Zeitpunkt wieder einsatzbereit. Der EPYC-Cluster läuft seit der Behebung des Problems um 17:25 CET wieder stabil, und der Vorfall konnte erfolgreich abgeschlossen werden.

Read the full incident report →

Started Jan 08, 2025, 09:21 PM UTC· resolved Jan 13, 2025, 02:33 AM UTC

1. identified Jan 08, 2025, 09:21 PM UTC

   Due to a problem in our internal network, one of the Ryzen 9 7950x(3D) clusters was destroyed.

2. identified Jan 09, 2025, 12:20 PM UTC

   We are currently waiting for failover hardware to move the KVM servers that are active on the host system to the failover host system so that we can reassemble the cluster.

3. identified Jan 09, 2025, 10:44 PM UTC

   The fallback host system has now been put into operation and is ready for the migration of the servers, which is now starting.

4. identified Jan 10, 2025, 03:52 PM UTC

   The rebuild of the Ryzen 9 7950x3D cluster has now begun. All KVM servers are now gradually being moved to the new cluster.

5. monitoring Jan 11, 2025, 12:48 PM UTC

   The last node has just been migrated to the new cluster. An update on further availability will follow in the next few hours.

6. resolved Jan 13, 2025, 02:33 AM UTC

   The migration has been closely monitored over the past few hours, and all systems are operating smoothly. The process is therefore considered successfully completed. Additionally, the cluster has been migrated to a separate VLAN to ensure that communication issues between the servers will no longer occur.

Read the full incident report →

Started Dec 28, 2024, 01:21 PM UTC· resolved Dec 28, 2024, 01:59 PM UTC

1. investigating Dec 28, 2024, 01:21 PM UTC

   Some KVM servers on Node epyc460 are currently experiencing network problems. We are already looking for the cause of the problem.

2. identified Dec 28, 2024, 01:40 PM UTC

   Problem has been found. We are working on a solution.

3. resolved Dec 28, 2024, 01:59 PM UTC

   Problem has been fixed. It was due to an incorrect DDoS protection setting.

Read the full incident report →

Started Dec 26, 2024, 01:58 AM UTC· resolved Dec 27, 2024, 02:07 PM UTC

1. identified Dec 26, 2024, 01:58 AM UTC

   Duplication of disks, which has led to a major problem in the management of KVM servers

2. monitoring Dec 26, 2024, 02:01 AM UTC

   After informing our customers in an email three days ago, we had to bring forward the maintenance work on epyc460 at short notice. All customers will be migrated to an emergency system so that we can complete the maintenance work without causing too much downtime.

3. monitoring Dec 26, 2024, 11:33 PM UTC

   The maintenance work on the host system has been completed. All VMs have now been migrated back.

4. resolved Dec 27, 2024, 02:07 PM UTC

   The maintenance work was successfully completed. All servers that were migrated to the xeon516 as a fallback host have been migrated back to epyc460.

Read the full incident report →