Farm HPC cluster incident
Campus wide power blip caused issues, nas-5-3 last holdout
Resolved after 2 days 1 hour, affecting Login and Storage and 1 more, on January 31, 2024.
nas-5-3 is automatically working its way through the last of the required work after the power surge, so everything should be back in service.
Affected components
Update timeline
- identified Jan 31, 2024, 05:32 PM UTC
At approximately 00:40 this morning (1/31) PG&E had a low voltage incident that affected campus. Most Farm devices were able to ride through the blip, but some NASii and some compute nodes were impacted. Admins are working on recovery now.
- identified Jan 31, 2024, 05:38 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 31, 2024, 08:11 PM UTC
Most NASii and nodes have been brought back into service. nas-5-3 is showing lingering issues so we are trying to get it back online. Additionally, some bm nodes have lost their ability to talk on the network. All VMs are offline as the NAS they use for storage is still impacted.
- monitoring Feb 01, 2024, 02:33 AM UTC
All systems except nas-5-3 have been fully returned to service at this point. nas-5-3 is currently in service, and we are monitoring it closely, but we may need to take it offline in the morning for a, hopefully, brief time. More news as it breaks.
- monitoring Feb 01, 2024, 07:52 PM UTC
nas-5-3 will be rebooted at 9am tomorrow (Friday) to verify some setting. The downtime should be brief, running jobs will pause and automatically resume when the NAS returns to service. Anyone with home directories on nas-5-3 will be unable to login or perform actions in their home during this time.
- resolved Feb 02, 2024, 06:51 PM UTC
nas-5-3 is automatically working its way through the last of the required work after the power surge, so everything should be back in service.
Your product runs on more than Farm HPC cluster
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services