Farm HPC cluster incident
nas-12-3 is suffering from extremely high load average
Resolved after 3 days 17 hours, affecting Storage, on November 13, 2023.
Deep investigation showed the root cause of the slowness was a large number of jobs doing an extremely high amount of writes. The server sustained 2 GB/sec, with peaks of 2.5 GB/sec of writes for 5 days. We are currently looking into tuning options to prevent that particular workload from starving out the more interactive workloads. Performance has returned to normal, and the user has been asked to run fewer simultaneous jobs. A separate project is in progress to replace drives that are approaching failure.
Affected components
Update timeline
- identified Nov 13, 2023, 11:31 PM UTC
nas-12-3 is suffering from extremely high load average which is causing performance issues. Administrators are investigating.
- identified Nov 14, 2023, 12:43 AM UTC
A bad disk has been identified as a possible culprit of the problem. Unfortunately, due to the hardware in use, removing the bad disk is a mini project and may require downtime of 2 NASii. Planning is in progress to determine the exact work required and a time to perform the work.
- resolved Nov 17, 2023, 04:41 PM UTC
Deep investigation showed the root cause of the slowness was a large number of jobs doing an extremely high amount of writes. The server sustained 2 GB/sec, with peaks of 2.5 GB/sec of writes for 5 days. We are currently looking into tuning options to prevent that particular workload from starving out the more interactive workloads. Performance has returned to normal, and the user has been asked to run fewer simultaneous jobs. A separate project is in progress to replace drives that are approaching failure.
Your product runs on more than Farm HPC cluster
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services