Farm HPC cluster incident

Farm login node having issues mounting some home and group directories

Major Resolved View vendor source →

Farm HPC cluster experienced a major incident on October 17, 2024 affecting Login, lasting 3h 31m. The incident has been resolved; the full update timeline is below.

Started
Oct 17, 2024, 04:49 PM UTC
Resolved
Oct 17, 2024, 08:20 PM UTC
Duration
3h 31m
Detected by Pingoru
Oct 17, 2024, 04:49 PM UTC

Affected components

Login

Update timeline

  1. identified Oct 17, 2024, 04:49 PM UTC

    Farm's login node is having issues mounting some NFS home and group directories. Admins are investigating.

  2. identified Oct 17, 2024, 04:57 PM UTC

    Recovery is going to require the login node to be rebooted, which will happen shortly. Any srun jobs will be killed, but sbatch jobs will continue without interruption.

  3. monitoring Oct 17, 2024, 06:31 PM UTC

    nas-4-1 was identified as the root cause for the Farm head node login issues. nas-4-1 was rebooted and appears to be serving correctly again. Admins are currently monitoring both systems.

  4. resolved Oct 17, 2024, 08:20 PM UTC

    This incident has been resolved.