DataFlair incident

ryzen461 and ryzen474 outage

Resolved after 4 days 5 hours, on January 8, 2025.

resolved Jan 13, 2025, 02:33 AM UTC

The migration has been closely monitored over the past few hours, and all systems are operating smoothly. The process is therefore considered successfully completed. Additionally, the cluster has been migrated to a separate VLAN to ensure that communication issues between the servers will no longer occur.

Started
Jan 08, 2025, 09:21 PM UTC
Resolved
Jan 13, 2025, 02:33 AM UTC
Duration
4d 5h
Detected by Pingoru
Jan 08, 2025, 09:21 PM UTC

Update timeline

  1. identified Jan 08, 2025, 09:21 PM UTC

    Due to a problem in our internal network, one of the Ryzen 9 7950x(3D) clusters was destroyed.

  2. identified Jan 09, 2025, 12:20 PM UTC

    We are currently waiting for failover hardware to move the KVM servers that are active on the host system to the failover host system so that we can reassemble the cluster.

  3. identified Jan 09, 2025, 10:44 PM UTC

    The fallback host system has now been put into operation and is ready for the migration of the servers, which is now starting.

  4. identified Jan 10, 2025, 03:52 PM UTC

    The rebuild of the Ryzen 9 7950x3D cluster has now begun. All KVM servers are now gradually being moved to the new cluster.

  5. monitoring Jan 11, 2025, 12:48 PM UTC

    The last node has just been migrated to the new cluster. An update on further availability will follow in the next few hours.

  6. resolved Jan 13, 2025, 02:33 AM UTC

    The migration has been closely monitored over the past few hours, and all systems are operating smoothly. The process is therefore considered successfully completed. Additionally, the cluster has been migrated to a separate VLAN to ensure that communication issues between the servers will no longer occur.

Your product runs on more than DataFlair

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →