Farm HPC cluster incident

nas-5-2 crashed

Resolved after 2 hours 51 minutes, affecting Storage, on March 27, 2025.

monitoring Mar 27, 2025, 04:39 PM UTC

nas-5-2 has been rebooted and verified to be back in service. It is taking a very high load of writes, so access will be sluggish until backed-up jobs catch up.

Started
Mar 27, 2025, 04:08 PM UTC
Resolved
Mar 27, 2025, 06:59 PM UTC
Duration
2h 51m
Detected by Pingoru
Mar 27, 2025, 04:08 PM UTC

Affected components

Storage

Update timeline

  1. identified Mar 27, 2025, 04:08 PM UTC

    nas-5-2 has crashed. Any home directories, or group directories, shared from there are currently hung. Admins are investigating.

  2. monitoring Mar 27, 2025, 04:39 PM UTC

    nas-5-2 has been rebooted and verified to be back in service. It is taking a very high load of writes, so access will be sluggish until backed-up jobs catch up.

  3. resolved Mar 27, 2025, 06:59 PM UTC

    nas-5-3 is once again correctly serving data.

Your product runs on more than Farm HPC cluster

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →