h8lio experienced a critical incident on April 5, 2025 affecting storage.h8l.io, lasting 7d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 05, 2025, 09:53 AM UTC
The Ceph disk 7 encountered some deep scrubbing issues. We are running use health check tools that may slow down the disk access.
- identified Apr 05, 2025, 04:30 PM UTC
Ceph cluster in global recovery. This may affect some services due to the un/mount pvc issues
- identified Apr 05, 2025, 09:53 PM UTC
New disks have been added to the cluster. The data load balancing is in progress (it may take few hours to be done).
- identified Apr 06, 2025, 06:59 AM UTC
The new Ceph OSDs (disks) have been rebalanced to insure the safety of the data. We are still working on the damaged placement groups.
- identified Apr 07, 2025, 02:37 AM UTC
Ceph is slowly recovering. We are making every effort to protect your data. We will contact you soon with secure recovery instructions.
- identified Apr 07, 2025, 04:04 AM UTC
While Ceph recovers, we initiate service restoration.
- identified Apr 07, 2025, 04:46 AM UTC
We managed to fix all Ceph Placement Group (PG) in error. There are still 74/391 PG which needs to be consolidated (deep subbed). Most of the services using PVC are still down for now. We are cleaning mount points to be able to restart them.
- resolved Apr 12, 2025, 10:01 AM UTC
This incident has been resolved.
- postmortem Apr 21, 2025, 05:43 PM UTC
An incident report will be soon provided. * Support email: [email protected] * Dedicated phone number: \+33.9.78.08.04.89 * [Persistent Volume Claim tools](https://resources.h8l.io/docs/kubernetes/volumes/#tools) * [Ticket system](https://sources.h8l.io/h8lio/h8lio/issues)