h8lio experienced a minor incident on July 24, 2024 affecting kube.h8l.io and storage.h8l.io and 1 more component, lasting 12h 20m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 24, 2024, 05:11 PM UTC
We are going to run several nodes check with potential interventions on them during the night. We will keep you posted on this thread.
- investigating Jul 24, 2024, 06:41 PM UTC
Full clusters check results: - Kubernetes: OK (minor network issues fixed) - Ceph:: WARN (we are deep checking the slow heartbeats on the node2's OSDs)
- monitoring Jul 24, 2024, 07:29 PM UTC
The slow latency OSD.1 (disk) has been removed from the Ceph cluster.
- resolved Jul 25, 2024, 05:31 AM UTC
This incident has been resolved.
- postmortem Jul 25, 2024, 07:19 AM UTC
The second node2 OSD disk failed this morning \(France\) and has been removed from the Ceph cluster at 08:30 CET. The data load balancing ended at 09:15 CET and Ceph cluster health is back to normal. We are still monitoring actively the node2 and the Ceph cluster.