Resolved after 4 days, affecting Slurm, on May 1, 2026.
Continued monitoring shows the workaround is still working, so this incident is resolved.
Affected components
Update timeline
- investigating May 01, 2026, 10:54 PM UTC
Admins are investigating an issue where most jobs on Farm are getting stuck in the completing stage.
- investigating May 02, 2026, 05:34 PM UTC
The issue is that slurmctld is not properly starting and completing jobs when under load. Admins have opened a Severity 1 ticket with SchedMD support and are waiting to hear back.
- monitoring May 05, 2026, 12:35 AM UTC
SchedMD identified the bug that was impacting Farm. The workaround they provided seems to have resolved the outage. Admins will continue to monitor.
- resolved May 05, 2026, 10:54 PM UTC
Continued monitoring shows the workaround is still working, so this incident is resolved.
Your product runs on more than Farm HPC cluster
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services