Resolved after 94 days 22 hours, affecting Login and Storage, on December 8, 2023.
Replacement drives have arrived are being swapped in to replace the failed drives.
Affected components
Update timeline
- investigating Dec 08, 2023, 05:32 PM UTC
nas-12-2 and nas-12-3 will be undergoing additional emergency maintenance to bring in more replacement hard drives. We will be pulling spares from nas-4-1 and nas-4-3, and there is a slight possibility of those NASii having temporary disk access issues. If this low probability event happens then jobs using that storage will pause, and should pick up with no issues when we are done. Downtime would be in the 5 to 15 minute range.
- monitoring Dec 11, 2023, 05:51 PM UTC
The spare drives were pulled from nas-4-3 without any issues, and were slotted into nas-12-[23] to continue the work of replacing failing drives. That replacement is still in progress. Admins are keeping a close eye on the progress, but the process of replacing the known bad drives has revealed other drives that are now having issues, so this will be a long process to bring nas-12-2 & 12-3 back to full disk redundancy.
- investigating Dec 12, 2023, 05:44 PM UTC
NAS-12-3 is currently unavailable. We are looking into the root cause.
- monitoring Dec 13, 2023, 12:11 AM UTC
Perrformance should be back to normal on NAS12-2 and NAS-12-3.
- resolved Mar 12, 2024, 04:24 PM UTC
Replacement drives have arrived are being swapped in to replace the failed drives.
Your product runs on more than Farm HPC cluster
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services