Stanford University Outage History
Stanford University is up right nowPingoru has recorded 5 incidents for Stanford University: 5 in the last 12 months and 2 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 19 of 720 (3%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 1 | 3 of 720 (0%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 1 | 51 of 744 (7%) |
| September 2026 | 1 | 19 of 720 (3%) |
| October 2026 | 0 | 0 of 585 (0%) |
Incidents, newest first
Minor September 25, 2026 Oak partial outage 19h 8m $OAK
Started Sep 25, 2026, 01:00 AM UTC· resolved Sep 25, 2026, 08:09 PM UTC
- investigating Sep 25, 2026, 01:00 AM UTC
We’re currently investigating a hardware issue on one of Oak’s storage enclosures. A subset of Oak paths may see brief slowness or intermittent I/O errors while we work through it.
- resolved Sep 25, 2026, 08:09 PM UTC
The affected hardware has been replaced and Oak is now fully operational again. Some files on Oak were inaccessible during this partial outage. Access has now been restored, but processes that were actively using those files may have remained stuck and will need to be cleared by restarting the compute nodes they were running on. As a result, several compute nodes are currently draining, which may delay the start of new jobs. Normal operation should resume once the currently running jobs terminate.
Minor August 16, 2026 Hardware issue on `$SCRATCH` 2d 3h $SCRATCH
Started Aug 16, 2026, 03:20 PM UTC· resolved Aug 18, 2026, 06:40 PM UTC
- investigating Aug 16, 2026, 03:20 PM UTC
A hardware-related failure occurred on the storage system that hosts $SCRATCH and $GROUP_SCRATCH. We’re investigating.
- monitoring Aug 16, 2026, 09:45 PM UTC
The issue has been identified and mitigated. Full resolution will require a hardware part replacement, which will likely take place tomorrow. In the meantime, the file system is up and running, and jobs should have resumed. We’ll monitor the situation until the hardware replacement is complete, and close this incident when it’s done.
- resolved Aug 18, 2026, 06:40 PM UTC
The failed component has been replaced, and the file system is now fully operational again.
Major April 19, 2026 Outage detected on status page 18h 47m Overall
Started Apr 19, 2026, 08:44 PM UTC· resolved Apr 20, 2026, 03:31 PM UTC
No vendor updates were posted for this incident.
Minor February 5, 2026 Investigating potential scheduling delays 14d 4h Slurm controller
Started Feb 05, 2026, 08:40 PM UTC· resolved Feb 20, 2026, 01:26 AM UTC
- investigating Feb 05, 2026, 08:40 PM UTC
We’re currently investigating some scheduling delays with jobs that have recently been submitted on Sherlock. Under certain circumstances, jobs may take longer to be dispatched and wait in queue for longer than usual. All jobs will eventually start, so we recommend keeping them in queue and to avoid cancelling jobs (re-submitting them later will only put them back at the end of the line). We’re working with the scheduler support and development teams on this incident, and will post updates when we have them.
- investigating Feb 07, 2026, 01:18 AM UTC
The scheduling delays are still being investigated. As mentioned initially, all jobs eventually get execute, so no action is required on the user part, besides a little bit more patience than usual. We’re aware of the trouble this may cause, and are working with the scheduler developers to identify the problem and find a path to resolution.
- identified Feb 11, 2026, 01:29 AM UTC
Work continues with the scheduler developers on this issue, and good progress is being made. A likely source of the scheduling delays has been identified, and we are now working on validating possible workarounds, before a fix can be developed, tested and deployed. As a reminder, all jobs will eventually start, so no action is required on your part. We appreciate your patience and will continue to post updates as we approach final resolution.
- monitoring Feb 17, 2026, 08:56 PM UTC
The root cause of the potential scheduling delays reported earlier has been identified as a bug that caused the job scheduler to make inefficient decisions on systems where many jobs request licenses (like Sherlock), resulting in jobs waiting longer than expected to start. The workaround currently in place has been validated, and scheduling is back to normal: no further delays are being observed. We are keeping this issue open until an official fix is released upstream and deployed on Sherlock.
- resolved Feb 20, 2026, 01:26 AM UTC
A fix addressing the root cause of the scheduling delays has been deployed. Job dispatch times have returned to normal, and the issue is now resolved. We appreciate users’ patience while we worked with the Slurm development team to identify and address the problem.