h8lio experienced a minor incident on August 5, 2025 affecting storage.h8l.io, lasting 1d 20h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 05, 2025, 07:24 PM UTC
Two OSDs (disks) are down. They are being replaced. You may experience slowdowns on "HDD" storage classes while data is being redistributed to the new drives.
- identified Aug 06, 2025, 03:20 AM UTC
The new Ceph host with 2 OSDs has been integrated to the cluster. The data rebalancing is in process.
- identified Aug 06, 2025, 06:49 AM UTC
The Ceph cluster is still recovering from the lost of the hdd osds.
- identified Aug 06, 2025, 08:16 AM UTC
The croit.io Ceph experts just informed us that we are experiencing the bug https://tracker.ceph.com/issues/70390 affecting our HDD disk pool. They are actively working to restore the service, but full recovery may take several hours.
- identified Aug 06, 2025, 12:57 PM UTC
One more OSD failed. We just received 2 new servers, each with with 2 OSDs. Their integration is in progress
- monitoring Aug 06, 2025, 08:50 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Aug 07, 2025, 05:49 AM UTC
The failing disks have been replaced and new hosts have been added. We are monitoring the Ceph cluster recovery which is still in progress.
- resolved Aug 07, 2025, 03:54 PM UTC
This incident has been resolved.
- postmortem Aug 11, 2025, 09:25 AM UTC
We would like to update you on a recent issue affecting our Ceph storage cluster. Igor Fedotov, reporter of [issue #70390](https://tracker.ceph.com/issues/70390), provided us through our recovery engagement with [croit.io](http://croit.io) the procedure to patch all Ceph OSDs. The issue originated during the July HDD migration. Although the Squid upgrade was performed in February, a feature affected by the known issue \(#70390\) was activated by default on the newly added HDD disks. This involved the `elastic_shared_blobs` parameter, which can impact OSD performance and stability. All OSDs in our Ceph cluster have now been patched. This involved globally disabling the `bluestore_elastic_shared_blobs` feature by setting it to `0` and recreating all existing OSDs to ensure their individual `elastic_shared_blobs` parameters were also set to `0`. The faulty new Squid feature has thus been fully disabled, which should prevent further occurrences of the issue. We continue to monitor cluster health and performance closely. We will keep you informed of any developments, including insights on why this issue, opened on March 11 with the highest priority \(“Immediate”\), remains open. If you have any questions or need further information, please contact our support team at [[email protected]](mailto:[email protected]).