Stanford University incident

Oak partial outage

Resolved after 19 hours 8 minutes, affecting $OAK, on September 25, 2026.

resolved Sep 25, 2026, 08:09 PM UTC

The affected hardware has been replaced and Oak is now fully operational again. Some files on Oak were inaccessible during this partial outage. Access has now been restored, but processes that were actively using those files may have remained stuck and will need to be cleared by restarting the compute nodes they were running on. As a result, several compute nodes are currently draining, which may delay the start of new jobs. Normal operation should resume once the currently running jobs terminate.

Started
Sep 25, 2026, 01:00 AM UTC
Resolved
Sep 25, 2026, 08:09 PM UTC
Duration
19h 8m
Detected by Pingoru
Sep 25, 2026, 01:00 AM UTC

Affected components

$OAK

Update timeline

  1. investigating Sep 25, 2026, 01:00 AM UTC

    We’re currently investigating a hardware issue on one of Oak’s storage enclosures. A subset of Oak paths may see brief slowness or intermittent I/O errors while we work through it.

  2. resolved Sep 25, 2026, 08:09 PM UTC

    The affected hardware has been replaced and Oak is now fully operational again. Some files on Oak were inaccessible during this partial outage. Access has now been restored, but processes that were actively using those files may have remained stuck and will need to be cleared by restarting the compute nodes they were running on. As a result, several compute nodes are currently draining, which may delay the start of new jobs. Normal operation should resume once the currently running jobs terminate.

Your product runs on more than Stanford University

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →