Farm HPC cluster experienced a major incident on January 6, 2026 affecting Storage, lasting 1d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 06, 2026, 08:25 PM UTC
nas-6-1 is still losing contact with its disks, which causes the ZFS file-system to suspend operation. Admins are investigating. This directly impacts the following PI group directories: darcantugrp3 dubcovskygrpBackup ejf4grp ejf4grp epontikegrp epontikegrp group group jmearlesgrp lmillergrp lmillergrp magertongrp magertongrp zhougrp zimanyigrp zimanyigrp This directly impacts the following users: afefm agoga amdiggs amnjoshi andresmr apjha caih chaiti cmhansen ddyoyo98 dguevara dgunruh ejf4 epontike eranario hkaman hzchris jbkeeler jmearles jyoun jyyi kertcher lilyb lion397 makovtun mevishal mhstan mmomayye momtanu mstigler nasiegel nbhagat nhaigh nikocj nparekh nwisuthi petersl pvraja rkdesai7 sonishiy spacemat vbgupta yuqing18 zfei zimanyi zmcrawfo ztzhao
- monitoring Jan 08, 2026, 12:41 AM UTC
Additional fixes have been put into place, and nas-6-1 currently working correctly again. Admin are monitoring the situation.
- resolved Jan 08, 2026, 07:13 PM UTC
nas-6-1 has been successfully serving data for 24 hours now, so it looks like the combination of updates resolved the issue it was experiencing.