Stanford University incident
`/scratch` file system is unresponsive
Stanford University experienced a minor incident on July 22, 2025 affecting $SCRATCH, lasting 2h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 22, 2025, 09:07 PM UTC
The /scratch filesystem (which serves $SCRATCH and $GROUP_SCRATCH) is experiencing some issues. Symptoms include hanging commands and non-responsive access when trying to access anything under /scratch. We’re currently investigating, and we’ll post updates as they become available
- investigating Jul 22, 2025, 09:48 PM UTC
We’re still investigating the issue, and are working on restoring access to /scratch as soon as possible.
- monitoring Jul 22, 2025, 10:51 PM UTC
The /scratch file system should be back up and running normally. Processes that were stuck on I/O should have resumed automatically, but in case applications reported explicit errors, feel free to resubmit those jobs or restart those processes. And to reach out to [email protected] if you have any questions.
- resolved Jul 22, 2025, 11:37 PM UTC
The issue has been resolved. We’ll keep an eye on things, but we’re confident this incident can be closed now.