Farm HPC cluster incident

nas-6-0 having issue

Major Resolved View vendor source →

Farm HPC cluster experienced a major incident on December 3, 2024 affecting Login and Storage, lasting 6d 20h. The incident has been resolved; the full update timeline is below.

Started
Dec 03, 2024, 11:20 PM UTC
Resolved
Dec 10, 2024, 07:57 PM UTC
Duration
6d 20h
Detected by Pingoru
Dec 03, 2024, 11:20 PM UTC

Affected components

LoginStorage

Update timeline

  1. identified Dec 03, 2024, 11:20 PM UTC

    nas-6-0 is currently having issues. It appears a single disk may be the culprit. Admins are trying to track down that disk to tell it to knock it off.

  2. monitoring Dec 04, 2024, 12:23 AM UTC

    Two drives were identified as causing issues in the ZFS pool. Both have been taken offline. Admins are still monitoring the situation.

  3. resolved Dec 10, 2024, 07:57 PM UTC

    The failing hard drives were indeed the source of the nas-6-0 ZFS outages. Replacements are on their way, but will not require downtime, so marking this incident as resolved.