Farm HPC cluster incident
Performance Degradation on Storage, Home Directories
Resolved after 48 days 4 hours, affecting Login and Storage, on October 23, 2024.
The downgraded kernel version appears to be avoiding the NFS bug. Admins have pinned a version that is not impacted by this bug.
Affected components
Update timeline
- investigating Oct 23, 2024, 03:57 PM UTC
System administrators are still seeing hung processes on Farm storage and are continuing to look into the root cause.
- identified Oct 23, 2024, 04:47 PM UTC
The same kernel bug on nas-4-1 has been identified as the root-cause. Admins are investigating the best path forward.
- identified Oct 23, 2024, 05:02 PM UTC
System administrators will be restarting NAS-4-1 storage and the Farm login node. Emergency downtime should be approximately one hour.
- monitoring Oct 23, 2024, 06:04 PM UTC
Both nas-4-1 and the login node have been downgraded to a kernel version that should not contain this bug. Admins are monitoring.
- identified Nov 04, 2024, 05:31 PM UTC
Unfortunately, the same problem reoccurred, so nas-4-1 and Farm's login node will be rebooted in a few minutes. Jobs running with sbatch should just pause and resume when the NFS server recovers. srun jobs will be killed when the login node is rebooted. Admins are still working to determine the combination of client and server kernel versions that is tickling this kernel bug.
- monitoring Nov 04, 2024, 06:26 PM UTC
The reboot of nas-4-1 seems to have cleared up the issues with the login node. At this point, a reboot of the login node does not appear to be required. Please open a ticket if you are still having problems.
- resolved Dec 10, 2024, 08:51 PM UTC
The downgraded kernel version appears to be avoiding the NFS bug. Admins have pinned a version that is not impacted by this bug.
Your product runs on more than Farm HPC cluster
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services