Farm HPC cluster Outage History
Farm HPC cluster is up right nowPingoru has recorded 51 incidents for Farm HPC cluster: 13 in the last 12 months and 2 in the last 90 days.
Incidents, newest first
Major August 27, 2026 Farm: partial outage 21h 23m Login, Proxmox Virtualization Nodes, Storage, high,low and 2 more
Started Aug 27, 2026, 12:03 AM UTC· resolved Aug 27, 2026, 09:27 PM UTC
- investigating Aug 27, 2026, 12:03 AM UTC
Monitoring has alerted us that a part of Farm is currently offline. This includes nas-4-0, nas-4-1, nas-4-2, nas-4-3, nas-5-2 and nas-5-3. Admins are onsite and investigating.
- investigating Aug 27, 2026, 12:05 AM UTC
We are continuing to investigate this issue.
- identified Aug 27, 2026, 12:18 AM UTC
One of the power breakers that powers parts of both Hive and Farm tripped. Data Center operators are calling facilities.
- identified Aug 27, 2026, 01:30 AM UTC
We are continuing to work on a fix for this issue.
- identified Aug 27, 2026, 02:13 AM UTC
Most NAS have been brought back into service. Nodes will remain down until we can reaccess power tomorrow.
- identified Aug 27, 2026, 04:33 PM UTC
Farm's login node has not recovered its NFS mounts and will be rebooted at 9:40am. Please save your work and log out.
- identified Aug 27, 2026, 04:57 PM UTC
The reboot of Farm's login node has allowed homedirs that were previously jammed to work again. Admins are still reviewing power usage before bringing nodes back online.
- resolved Aug 27, 2026, 09:27 PM UTC
All nodes and systems have been brought back into service.
Critical July 5, 2026 Quobyte is down due to another partial power failure in the Data Center 1d 7h Storage
Started Jul 05, 2026, 05:19 PM UTC· resolved Jul 07, 2026, 12:19 AM UTC
- identified Jul 05, 2026, 05:19 PM UTC
Power has partially failed in the Data Center, which has brought Quobyte down again. Non-Quoybte (i.e., NFS storage mounted in /group/) is still working correctly. Staff are in contact with the Data Center Operators, and Facilities is being scheduled to investigate the power issue.
- identified Jul 05, 2026, 10:32 PM UTC
Admins are on-site waiting for campus facilities.
- identified Jul 06, 2026, 04:00 PM UTC
We have brought the Quobyte file system up and are working on rebuilding data volumes.
- resolved Jul 07, 2026, 12:19 AM UTC
Admins have brought Quobyte back online after the partial Data Center power outage.
Major June 28, 2026 Farm's nas-12-3 having disk issues 2d 4h Storage
Started Jun 28, 2026, 05:25 PM UTC· resolved Jun 30, 2026, 09:49 PM UTC
- identified Jun 28, 2026, 05:25 PM UTC
Farm's nas-12-3 is having critical disk issues and is currently unavailable. This is impacting all users with home directories still in the adamgrp legacy free-tier storage as well as the expired emoniergrp and spatial03 shares. Admins are evaluating options.
- resolved Jun 30, 2026, 09:49 PM UTC
nas-12-3 managed to rebuild data onto the new drives over the weekend, so it is no longer teetering on the brink of catastrophic data loss. At this point it is fully back in service and will be closely monitored for additional issues.
Major June 26, 2026 Storage Mounts on Farm are unavailable 2d Login, Storage
Started Jun 26, 2026, 04:35 PM UTC· resolved Jun 28, 2026, 05:05 PM UTC
- identified Jun 26, 2026, 04:35 PM UTC
Some storage mounts, including Quobyte mounts, are not accessible from the Farm login node and some compute. System administrators will need to reboot the login node.
- resolved Jun 28, 2026, 05:05 PM UTC
These storage mounts became operational when the login node was rebooted.
Major June 24, 2026 Power outage at DC is causing Quobyte File Storage to be inaccessible. 19h 27m Storage
Started Jun 24, 2026, 12:10 AM UTC· resolved Jun 24, 2026, 07:37 PM UTC
- investigating Jun 24, 2026, 12:10 AM UTC
Facilities has been called to reset the breaker that powers the Quobyte clustered storage. Data Center operators have been unable to restore power.
- resolved Jun 24, 2026, 07:37 PM UTC
Power has been restored to the data center row. A breaker went bad in the power panel. System administrators have restored Quobyte to full functionality.
Critical June 23, 2026 Home directories not automounting on nodes from Slurm jobs 50m high,low, bmh,bmm, bgpu, gpuh,gpum
Started Jun 23, 2026, 06:22 PM UTC· resolved Jun 23, 2026, 07:13 PM UTC
- identified Jun 23, 2026, 06:22 PM UTC
Farm nodes are not correctly automounting user's home directories when new jobs are submitted to that node. This causes the launched jobs to fail. Admins are investigating.
- resolved Jun 23, 2026, 07:13 PM UTC
A configuration change for Slurm has been implemented that resolves this issue. All user home directories will now mount correctly within Slurm jobs.
Critical May 1, 2026 Jobs getting stuck in completing 4d Slurm
Started May 01, 2026, 10:54 PM UTC· resolved May 05, 2026, 10:54 PM UTC
- investigating May 01, 2026, 10:54 PM UTC
Admins are investigating an issue where most jobs on Farm are getting stuck in the completing stage.
- investigating May 02, 2026, 05:34 PM UTC
The issue is that slurmctld is not properly starting and completing jobs when under load. Admins have opened a Severity 1 ticket with SchedMD support and are waiting to hear back.
- monitoring May 05, 2026, 12:35 AM UTC
SchedMD identified the bug that was impacting Farm. The workaround they provided seems to have resolved the outage. Admins will continue to monitor.
- resolved May 05, 2026, 10:54 PM UTC
Continued monitoring shows the workaround is still working, so this incident is resolved.
Major March 19, 2026 Partial degradation of logins to Farm 3h 14m Login
Started Mar 19, 2026, 05:40 PM UTC· resolved Mar 19, 2026, 08:55 PM UTC
- identified Mar 19, 2026, 05:40 PM UTC
The ability to log in to Farm is currently flapping for some users. The underlying issue has been identified, and a remedy is being worked on.
- resolved Mar 19, 2026, 08:55 PM UTC
This incident has been resolved.
Notice February 24, 2026 CSE subdomain offline. Farm available at farm.hpc.ucdavis.edu. 1d 4h Login
Started Feb 24, 2026, 07:31 PM UTC· resolved Feb 26, 2026, 12:25 AM UTC
- investigating Feb 24, 2026, 07:31 PM UTC
Farm is no longer available at farm.cse.ucdavis.edu, but IS available through farm.hpc.ucdavis.edu. System administrators are looking into the farm.cse DNS record availability. Please use farm.hpc.ucdavis.edu to log into Farm.
- resolved Feb 26, 2026, 12:25 AM UTC
Updated infrastructure was put in place to support farm.cse.ucdavis.edu. This is an old name. Please start using the new name: farm.hpc.ucdavis.edu
Critical January 20, 2026 Slurm not available 56m Slurm
Started Jan 20, 2026, 04:32 PM UTC· resolved Jan 20, 2026, 05:28 PM UTC
- investigating Jan 20, 2026, 04:32 PM UTC
Slurm is not able to launch or view jobs. System administrators are currently looking into the issue.
- resolved Jan 20, 2026, 05:28 PM UTC
Misconfigured Slurm entries caused the job scheduler to not start. System administrators fixed the issue.
Major January 6, 2026 nas-6-1 still not functioning 1d 22h Storage
Started Jan 06, 2026, 08:25 PM UTC· resolved Jan 08, 2026, 07:13 PM UTC
- investigating Jan 06, 2026, 08:25 PM UTC
nas-6-1 is still losing contact with its disks, which causes the ZFS file-system to suspend operation. Admins are investigating. This directly impacts the following PI group directories: darcantugrp3 dubcovskygrpBackup ejf4grp ejf4grp epontikegrp epontikegrp group group jmearlesgrp lmillergrp lmillergrp magertongrp magertongrp zhougrp zimanyigrp zimanyigrp This directly impacts the following users: afefm agoga amdiggs amnjoshi andresmr apjha caih chaiti cmhansen ddyoyo98 dguevara dgunruh ejf4 epontike eranario hkaman hzchris jbkeeler jmearles jyoun jyyi kertcher lilyb lion397 makovtun mevishal mhstan mmomayye momtanu mstigler nasiegel nbhagat nhaigh nikocj nparekh nwisuthi petersl pvraja rkdesai7 sonishiy spacemat vbgupta yuqing18 zfei zimanyi zmcrawfo ztzhao
- monitoring Jan 08, 2026, 12:41 AM UTC
Additional fixes have been put into place, and nas-6-1 currently working correctly again. Admin are monitoring the situation.
- resolved Jan 08, 2026, 07:13 PM UTC
nas-6-1 has been successfully serving data for 24 hours now, so it looks like the combination of updates resolved the issue it was experiencing.
Major October 30, 2025 Quobyte Storage is Unavailable 22d Storage
Started Oct 30, 2025, 05:42 PM UTC· resolved Nov 21, 2025, 06:33 PM UTC
- investigating Oct 30, 2025, 05:42 PM UTC
Quobyte clustered storage is currently unavailable on Farm. System Administrators are currently investigating the issue.
- monitoring Oct 30, 2025, 06:26 PM UTC
System administrators have made the Quobyte storage available again. The issue seemed to be related to a hanging file-system mount; system administrators will attempt to diagnose a root cause while monitoring continued file system access.
- resolved Nov 21, 2025, 06:33 PM UTC
This incident has been resolved.
Critical September 23, 2025 Farm not currently accepting logins 44m Login
Started Sep 23, 2025, 12:43 AM UTC· resolved Sep 23, 2025, 01:28 AM UTC
- investigating Sep 23, 2025, 12:43 AM UTC
Due to a PAM configuration issue, the ability to log in to Farm is currently broken. Admins are working to resolve the issue.
- resolved Sep 23, 2025, 01:28 AM UTC
Farm's login issue has been resolved, and users can log in again.
Notice July 25, 2025 Quobyte Unavaiable 3d 3h Storage
Started Jul 25, 2025, 05:18 PM UTC· resolved Jul 28, 2025, 08:25 PM UTC
- identified Jul 25, 2025, 05:18 PM UTC
Quobyte storage is unavailable. We have identified the problem and are working to return service.
- monitoring Jul 25, 2025, 05:31 PM UTC
Quobyte storage is available again. We are continuing to monitor.
- resolved Jul 28, 2025, 08:25 PM UTC
This incident has been resolved.
Major July 23, 2025 SSH availability 6h 16m Login
Started Jul 23, 2025, 03:08 PM UTC· resolved Jul 23, 2025, 09:25 PM UTC
- investigating Jul 23, 2025, 03:08 PM UTC
Some users report not being able to log into Farm's head node, but they can use Open OnDemand. System administrators are currently looking into the issue.
- monitoring Jul 23, 2025, 04:10 PM UTC
System administrators have fixed an issue with the account synchronization process and will continue to monitor the database for inconsistencies. Users should now be able to log in without issue.
- resolved Jul 23, 2025, 09:25 PM UTC
This issue has been resolved; system administrators have fixed replication issues with the account provisioning process.
Minor June 16, 2025 Datacenter Maintenance 7d Login, Proxmox Virtualization Nodes, Storage, high,low and 6 more
Started Jun 16, 2025, 04:00 PM UTC· resolved Jun 23, 2025, 04:38 PM UTC
- identified Jun 16, 2025, 04:00 PM UTC
Bi-annual maintenance due to data center generator test and upgrades.
- resolved Jun 23, 2025, 04:38 PM UTC
The maintenance is done and all the services are back.
Major April 24, 2025 Farm's slurmdbd having intermittent issues 152d Slurm
Started Apr 24, 2025, 12:22 AM UTC· resolved Sep 23, 2025, 12:42 AM UTC
- investigating Apr 24, 2025, 12:22 AM UTC
Farm's slurmdbd is having intermittent issues. If you see an error like below, it means the problem has occurred again, and we will restart slurmdbd to bring it back into service. """sacctmgr: error: _open_persist_conn: failed to open persistent connection to host:monitoring-ib:6819: Connection timed out sacctmgr: error: Sending PersistInit msg: Connection timed out""" We have a support case open with SchedMD and will update this issue as we learn more.
- resolved Sep 23, 2025, 12:42 AM UTC
This incident has been resolved.
Major April 1, 2025 Farm: nas-12-2 down due to multiple disk failures 22d 23h Login, Storage
Started Apr 01, 2025, 01:09 AM UTC· resolved Apr 24, 2025, 12:19 AM UTC
- identified Apr 01, 2025, 01:09 AM UTC
nas-12-2 has suffered from multiple disk failures. Admins are investigating the best path forward. The following group directories are currently unavailable: awhitehegrp millermrgrp millsgrp runciegrp weimergrp yujingrp The following home directories are unavailable: aavalos7 awhitehe barao bcbaikie bcweimer berdeja crice crios cschles dglemay djprince dkblaufu drbandoy eabernat ecgranad edkoch emmaluu eoziolor fengq hahudson hemstrow hxhu jagill jajpark jamcgirr jassim jcariute jdowen jenwash jmiller1 jroach jrwashab jxnliu katng23 ljcohen madarm11 mam12n mary363 millermr mlyjones mmosmond motch mtreiber namcnabb nmariano nreid pjseba profeta prvasque psbapat rsbrenna sakre saumyaw scsastry seboles sejoslin smhigdon spatel23 tmbolt vfbetsis vpdunne wolfie12 xmixu yoxue ytakim ywdong
- identified Apr 02, 2025, 08:59 PM UTC
Summary: nas-12-2 could be online Friday at the earliest, but more likely early next week. In consultation with Adam Getchell, the decision has been made to do low-level disk copy from the old, failing drives, to new drives. This will minimize the potential for data loss. This process is expected to finish Thursday at the earliest. Subsequently, the new disks will be added back to the ZFS pool, and we will trigger a full ZFS data scrub. When that finishes, we will know exactly how much, if any, data loss there is and which files are impacted. That data scrub will take a minimum of 24 hours, so the earliest nas-12-2 could be back in service is late Friday. It is more likely the scrub will run through the weekend, so a more realistic return-to-service is early next week.
- monitoring Apr 04, 2025, 09:31 PM UTC
As tends to happen with failing hard drives, data recovery often goes slower than hoped. Two drives had 100% of the data recovered, and a third had 99.99% recovered. The last drive failed too hard to recover data from, but that is okay, ZFS should be able to reconstruct everything it needs from the first three. A ZFS scrub (data verification) is in progress. When this finishes, likely early next week, we will know for sure the state of all the data on nas-12-2.
- identified Apr 05, 2025, 12:47 AM UTC
The ZFS scrub has been started and is being watched carefully.
- identified Apr 07, 2025, 09:49 PM UTC
The ZFS pool scrub (data verification) is in progress. As you can imagine, 409 TB of data takes a while to verify. The current ETA is that it will finish sometime late tonight. This scrub has caused 3 additional hard drives to drop out. The executive decision has been made to replace those drives before allowing users to access the pool. The estimate is an additional 3 days for those drives to have all the data reconstructed onto them, so our best-guess ETA for return-to-service is late this week. We will post additional updates as the disk replacement proceeds.
- identified Apr 09, 2025, 05:28 PM UTC
Disk replacements were successfully performed yesterday, and data reconstruction onto them is in progress. ZFS is currently estimating it will finish in a little over three days, so the best-case estimate is nas-12-2 will be available for use late Saturday evening. We will provide more updates as the reconstruction progresses.
- monitoring Apr 11, 2025, 11:10 PM UTC
nas-12-2's resilver finished ahead of schedule, so we have re-enabled it for use in Farm. One disk kicked off another resilver, but it is the only disk in that vdev with any issues, so we feel pretty comfortable allowing that to happen in the background. If you run into issues with nas-12-2, please open a Farm Support Ticket.
- resolved Apr 24, 2025, 12:19 AM UTC
nas-12-2 recovery was successful. We were able to scrape enough data from the failing drives that ZFS was able to rebuild onto new drives.
Major March 27, 2025 nas-5-2 crashed 2h 51m Storage
Started Mar 27, 2025, 04:08 PM UTC· resolved Mar 27, 2025, 06:59 PM UTC
- identified Mar 27, 2025, 04:08 PM UTC
nas-5-2 has crashed. Any home directories, or group directories, shared from there are currently hung. Admins are investigating.
- monitoring Mar 27, 2025, 04:39 PM UTC
nas-5-2 has been rebooted and verified to be back in service. It is taking a very high load of writes, so access will be sluggish until backed-up jobs catch up.
- resolved Mar 27, 2025, 06:59 PM UTC
nas-5-3 is once again correctly serving data.
Minor March 13, 2025 Farm's nas-6-1 having issues 22h 18m Login, Storage
Started Mar 13, 2025, 06:37 PM UTC· resolved Mar 14, 2025, 04:55 PM UTC
- investigating Mar 13, 2025, 06:37 PM UTC
Farm's nas-6-1 is currently having issues. Admins are investigating.
- monitoring Mar 13, 2025, 07:41 PM UTC
A faulty hard drive has been identified as the most likely cause of the NAS outage. This drive has been forced offline and admins are monitoring.
- monitoring Mar 13, 2025, 07:42 PM UTC
We are continuing to monitor for any further issues.
- resolved Mar 14, 2025, 04:55 PM UTC
Continued monitoring shows that disabling the broken hard drive has resolved the issue.
Major March 12, 2025 Data Center power blip 2h 2m Storage, high,low, bmh,bmm, bgpu and 1 more
Started Mar 12, 2025, 03:56 PM UTC· resolved Mar 12, 2025, 05:59 PM UTC
- identified Mar 12, 2025, 03:57 PM UTC
The Data Center suffered a power blip this morning. Staff are onsite and investigating.
- resolved Mar 12, 2025, 05:59 PM UTC
All Farm compute nodes and storage impacted have been brought back into service.
Major February 26, 2025 Farm login issues 44m Login
Started Feb 26, 2025, 12:05 AM UTC· resolved Feb 26, 2025, 12:49 AM UTC
- investigating Feb 26, 2025, 12:05 AM UTC
Farm admins are investigating a login issue.
- resolved Feb 26, 2025, 12:49 AM UTC
The issue has been identified and resolved. Additionally, the root-cause has been identified, and a workaround put in place that should prevent it from reoccurring.
Minor February 19, 2025 Some group and home directories unable to mount on the login node 1d 3h Login
Started Feb 19, 2025, 10:26 PM UTC· resolved Feb 21, 2025, 01:35 AM UTC
- identified Feb 19, 2025, 10:26 PM UTC
Some group and home directories are unable to mount on the login node. An emergency reboot of the login node is scheduled for 5pm tomorrow (Thursday). This will not impact any sbatch jobs, though it will cause all srun jobs to fail.
- resolved Feb 21, 2025, 01:35 AM UTC
This incident has been resolved.
Major January 7, 2025 Various software, including RStudio, still broken following maintenance window 43d 3h Software
Started Jan 07, 2025, 06:32 PM UTC· resolved Feb 19, 2025, 10:19 PM UTC
- identified Jan 07, 2025, 06:32 PM UTC
Sysadmins are actively working to restore software that is no longer working since the maintenance window. This includes, but is not limited to: * RStudio (which also breaks Rstudio Server in OnDemand) * Maker * Braker * BUSCO * Spades * multiqc * nvhpc Fixed: * The 52 pre-built conda environments * Julia * apptainer * OpenMPI (version 4.1.5)
- resolved Feb 19, 2025, 10:19 PM UTC
This incident has been resolved.
Major December 3, 2024 nas-6-0 having issue 6d 20h Login, Storage
Started Dec 03, 2024, 11:20 PM UTC· resolved Dec 10, 2024, 07:57 PM UTC
- identified Dec 03, 2024, 11:20 PM UTC
nas-6-0 is currently having issues. It appears a single disk may be the culprit. Admins are trying to track down that disk to tell it to knock it off.
- monitoring Dec 04, 2024, 12:23 AM UTC
Two drives were identified as causing issues in the ZFS pool. Both have been taken offline. Admins are still monitoring the situation.
- resolved Dec 10, 2024, 07:57 PM UTC
The failing hard drives were indeed the source of the nas-6-0 ZFS outages. Replacements are on their way, but will not require downtime, so marking this incident as resolved.