Farm HPC cluster incident

nas-6-0 currently offline

Resolved after 1 day 22 hours, affecting Storage, on October 23, 2023.

resolved Oct 25, 2023, 03:25 PM UTC

It appears the failing drive was the root cause of the issue. With that drive out of service, the pool has returned to normal.

Started
Oct 23, 2023, 04:40 PM UTC
Resolved
Oct 25, 2023, 03:25 PM UTC
Duration
1d 22h
Detected by Pingoru
Oct 23, 2023, 04:40 PM UTC

Affected components

Storage

Update timeline

  1. identified Oct 23, 2023, 04:40 PM UTC

    nas-6-0 is once again having issues and will be down for a little while so we can do a deeper investigation.

  2. monitoring Oct 23, 2023, 06:01 PM UTC

    nas-6-0 has been brought back into service and is being closely monitored.

  3. investigating Oct 23, 2023, 08:25 PM UTC

    nsa-6-0 is again having issues talking to the disks in the JBOD and I/O is suspended. We are investigating.

  4. monitoring Oct 23, 2023, 08:38 PM UTC

    A bad drive has been identified and has been forced out of service. We are, again, monitoring the system.

  5. resolved Oct 25, 2023, 03:25 PM UTC

    It appears the failing drive was the root cause of the issue. With that drive out of service, the pool has returned to normal.

Your product runs on more than Farm HPC cluster

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →