Farm HPC cluster incident
Farm's login node suffering because of an issue with nas-6-0
Resolved after 3 hours 24 minutes, affecting Login and Storage, on March 22, 2024.
Root cause was the badly failed hard drive. After rebooting both systems everything looks to be behaving correctly again.
Affected components
Update timeline
- investigating Mar 22, 2024, 07:36 PM UTC
Farm's login node is suffering and responding extremely slowly. Investigations are under way.
- identified Mar 22, 2024, 07:51 PM UTC
nas-6-0 is having disk issues, which is causing IO to jam on the head node. Admins are investigating.
- identified Mar 22, 2024, 08:03 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Mar 22, 2024, 09:55 PM UTC
One disk on nas-6-0 was misbehaving badly enough to completely ruin I/O. We removed the drive, but that was not enough to fully restore functionality. nas-6-0 has been rebooted and is being monitored.
- monitoring Mar 22, 2024, 10:23 PM UTC
Fallout from the nas-6-0 issue is causing a problem on the head node of a type that the sysadmins have never seen. A reboot of the head node will be happening shortly.
- resolved Mar 22, 2024, 11:00 PM UTC
Root cause was the badly failed hard drive. After rebooting both systems everything looks to be behaving correctly again.
Your product runs on more than Farm HPC cluster
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services