h8lio incident

[ceph] recovery throughput

Minor Resolved View vendor source →

h8lio experienced a minor incident on July 25, 2023 affecting storage.h8l.io, lasting 3h 5m. The incident has been resolved; the full update timeline is below.

Started
Jul 25, 2023, 08:06 AM UTC
Resolved
Jul 25, 2023, 11:11 AM UTC
Duration
3h 5m
Detected by Pingoru
Jul 25, 2023, 08:06 AM UTC

Affected components

storage.h8l.io

Update timeline

  1. investigating Jul 25, 2023, 08:06 AM UTC

    A recovery throughput on the data cluster Ceph has been noticed since last night. We are investigatin the cause. You may experience some slow connections (even timeout) on your standard storage class persistent volume claims. Monitoring merics and Spinnaker pipeline save issues have been fixed.

  2. identified Jul 25, 2023, 09:38 AM UTC

    The issue has been identified. We are working to clean up the unattached persistent volumes.

  3. resolved Jul 25, 2023, 11:11 AM UTC

    Orphan persistent volumes cleaned up. We are going to update this night intervention to finish the applied solution validation process