Farm HPC cluster incident
Farm Jobs stuck in SUSPENDED, COMPLETING states
Resolved after 7 hours 55 minutes, affecting Slurm, on September 4, 2024.
Rebooting nas-5-2 so clients could mount data from it has cleared up the issue. Additionally, most of the hundreds of jobs stuck in COMPLETING (CG) have finished. The remaining 3 jobs stuck in COMPLETING should clear up as those nodes are drained and able to be rebooted.
Affected components
Update timeline
- investigating Sep 04, 2024, 03:55 PM UTC
We are experiencing issues where jobs on Farm are becoming suspended when no suspense should occur. Jobs are also being stuck in 'COMPLETING' when they should not be. We are currently looking into a root cause.
- investigating Sep 04, 2024, 06:04 PM UTC
Farm is currently unable to mount NFS on many nodes, including the system where the Slurm controller runs. Admins are investigating, but Farm may be unstable and/or not act as usual during this investigation.
- investigating Sep 04, 2024, 06:48 PM UTC
One NAS (nas-5-2) has been identified as hanging when new mounts were attempted from it. It is currently being rebooted and will be checked closely after it finishes.
- monitoring Sep 04, 2024, 07:33 PM UTC
Rebooting nas-5-2 allowed new mounts from it, so the cluster should be recovering. Admins are still monitoring to make sure it keeps working correctly.
- resolved Sep 04, 2024, 11:51 PM UTC
Rebooting nas-5-2 so clients could mount data from it has cleared up the issue. Additionally, most of the hundreds of jobs stuck in COMPLETING (CG) have finished. The remaining 3 jobs stuck in COMPLETING should clear up as those nodes are drained and able to be rebooted.
Your product runs on more than Farm HPC cluster
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services