h8lio incident

[ceph] hdd disk failure recovery

Minor Resolved View vendor source →

h8lio experienced a minor incident on September 14, 2023 affecting storage.h8l.io and monitoring.h8l.io, lasting 20h 11m. The incident has been resolved; the full update timeline is below.

Started
Sep 14, 2023, 10:23 AM UTC
Resolved
Sep 15, 2023, 06:35 AM UTC
Duration
20h 11m
Detected by Pingoru
Sep 14, 2023, 10:23 AM UTC

Affected components

storage.h8l.iomonitoring.h8l.io

Update timeline

  1. monitoring Sep 14, 2023, 10:23 AM UTC

    Standard storage latency during data balancing after disk failure. No impact on data.

  2. monitoring Sep 14, 2023, 04:46 PM UTC

    We are continuing to monitor for any further issues.

  3. monitoring Sep 15, 2023, 06:18 AM UTC

    We are continuing to monitor for any further issues.

  4. resolved Sep 15, 2023, 06:35 AM UTC

    The faulty disk has been replaced during the night. The standard storage class is back to normal.