DataFlair experienced a major incident on June 23, 2025, lasting 5h 38m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Jun 23, 2025, 02:29 PM UTC
Our storage system ryzen5900X-1 encountered an error during its disk analysis. It incorrectly detected the storage as only half full, while in reality, the system had already reached full capacity. This misjudgment has now caused an overflow situation. We are currently performing a thorough check of all disks and investigating the underlying cause. We’ll provide updates as soon as we have more information.
- investigating Jun 23, 2025, 02:31 PM UTC
We have found the error. Some of the virtual discs were created in RAW format and caused the hard drives to overflow.
- monitoring Jun 23, 2025, 02:38 PM UTC
We now migrate the RAW discs temporarily to another disc in order to be able to restore them as a virt disc. There is no loss of data.
- investigating Jun 23, 2025, 02:39 PM UTC
The first servers have now been migrated.
- monitoring Jun 23, 2025, 07:57 PM UTC
The migration to our external storage to clean up the storage system has been completed. Individual VMs are now starting again and the others are being migrated back in the correct format.
- resolved Jun 23, 2025, 08:07 PM UTC
All VMs are now running again. The disc migration is running in the background.