Farm HPC cluster incident

Farm: partial outage

Resolved after 21 hours 23 minutes, affecting Login and Proxmox Virtualization Nodes and 1 more, on August 27, 2026.

identified Aug 27, 2026, 04:57 PM UTC

The reboot of Farm's login node has allowed homedirs that were previously jammed to work again. Admins are still reviewing power usage before bringing nodes back online.

Started
Aug 27, 2026, 12:03 AM UTC
Resolved
Aug 27, 2026, 09:27 PM UTC
Duration
21h 23m
Detected by Pingoru
Aug 27, 2026, 12:03 AM UTC

Affected components

LoginProxmox Virtualization NodesStoragehigh,lowbmh,bmmgpuh,gpum

Update timeline

  1. investigating Aug 27, 2026, 12:03 AM UTC

    Monitoring has alerted us that a part of Farm is currently offline. This includes nas-4-0, nas-4-1, nas-4-2, nas-4-3, nas-5-2 and nas-5-3. Admins are onsite and investigating.

  2. investigating Aug 27, 2026, 12:05 AM UTC

    We are continuing to investigate this issue.

  3. identified Aug 27, 2026, 12:18 AM UTC

    One of the power breakers that powers parts of both Hive and Farm tripped. Data Center operators are calling facilities.

  4. identified Aug 27, 2026, 01:30 AM UTC

    We are continuing to work on a fix for this issue.

  5. identified Aug 27, 2026, 02:13 AM UTC

    Most NAS have been brought back into service. Nodes will remain down until we can reaccess power tomorrow.

  6. identified Aug 27, 2026, 04:33 PM UTC

    Farm's login node has not recovered its NFS mounts and will be rebooted at 9:40am. Please save your work and log out.

  7. identified Aug 27, 2026, 04:57 PM UTC

    The reboot of Farm's login node has allowed homedirs that were previously jammed to work again. Admins are still reviewing power usage before bringing nodes back online.

  8. resolved Aug 27, 2026, 09:27 PM UTC

    All nodes and systems have been brought back into service.

Your product runs on more than Farm HPC cluster

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →