Skytap experienced a major incident on February 11, 2025 affecting US-East-2 Storage, lasting 9h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 11, 2025, 12:25 PM UTC
We are currently investigating alerts related to a possible storage issue in our US-East-2 Region. We are actively investigating these reports, and will provide further details regarding symptoms and scope within one hour.
- identified Feb 11, 2025, 12:57 PM UTC
We have identified a storage issue in our US-East-2 Region. Our Engineers are actively working on this issue with high priority. Symptoms for this issue may include: - Inability of some Virtual Machines in this region to change state (ie, run, suspend, shutdown) - Degraded disk read/write performance for some running Virtual Machines in this region - Deploying from templates in this region may fail - Saving templates in this region may fail - Copying environments or templates out of this region may fail - VM Imports and Exports out of this region may fail We will provide an additional update within one hour.
- identified Feb 11, 2025, 01:54 PM UTC
We have identified a storage issue in our US-East-2 Region. Our Engineers are actively working on this issue with high priority. Symptoms for this issue may include: - Inability of some Virtual Machines in this region to change state (ie, run, suspend, shutdown) - Some Virtual Machines in this region may become unresponsive or hung state. - Degraded disk read/write performance for some running Virtual Machines in this region - Deploying from templates in this region may fail - Saving templates in this region may fail - Copying environments or templates out of this region may fail - VM Imports and Exports out of this region may fail We will provide an additional update as soon as there is a significant change in status.
- identified Feb 11, 2025, 04:58 PM UTC
We have identified an issue with one of our storage subsystems in our US-East-2 Region. Our Engineers are continuing active work on this issue with the highest priority. Symptoms for this issue may include: - Inability of some Virtual Machines in this region to change state (ie, run, suspend, shutdown) - Some Virtual Machines in this region may become unresponsive or hung state. - Degraded disk read/write performance for some running Virtual Machines in this region - Deploying from templates in this region may fail - Saving templates in this region may fail - Copying environments or templates out of this region may fail - VM Imports and Exports out of this region may fail We will provide additional updates as soon as there is a significant change in status.
- monitoring Feb 11, 2025, 08:51 PM UTC
We have made changes to alleviate an issue with one storage sub-system in our US-East-2 region. The following symptoms should now be resolved. If you continue to experience any of the following symptoms, please contact Skytap Support ([email protected]) for immediate assistance: - Inability of some Virtual Machines in this region to change state (ie, run, suspend, shutdown) - Degraded disk read/write performance for some running Virtual Machines in this region - Deploying from templates in this region may fail - Saving templates in this region may fail - Copying environments or templates out of this region may fail We will provide an additional update within one hour.
- resolved Feb 11, 2025, 09:52 PM UTC
The storage issue in our US-East-2 Region has now been resolved. The following symptoms should now be alleviated: - Inability of some Virtual Machines in this region to change state (ie, run, suspend, shutdown) - Degraded disk read/write performance for some running Virtual Machines in this region - Deploying from templates in this region may fail - Saving templates in this region may fail - Copying environments or templates out of this region may fail This disruption may have resulted in some running Virtual Machines being disconnected from their storage for a period of time. Affected VM owners have been notified directly. If you have a VM reporting an error or issue, or if you continue to experience any of the symptoms mentioned above, please contact Skytap Support ([email protected]) for immediate assistance.