Velocity Host experienced a major incident on July 12, 2020 affecting (vh) Public vCloud, lasting 3h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 12, 2020, 08:27 PM UTC
At 3:50 am this morning engineers noticed a host in our CLOUD Cluster experiencing vSAN stability issues. Host ESX 08 stalled and was unresponsive, VMware HA did not kick in and a manual power cycle was required forcing the host to release VM's so they could be (vMotioned) moved to other available hosts. Engineers are working restoring VM's to other hosts. The host is currently pending VMware review. A postmortem will be supplied once logs are collected and reviewed by VMware to identify the root cause of the issue. Thank you (vh) Support
- investigating Jul 12, 2020, 09:12 PM UTC
Status Update All VM's have been restored to new hosts. We are currently implementing a plan to shift all workloads off vSan to other available storage. Thank you. (vh) Support
- investigating Jul 12, 2020, 09:12 PM UTC
We are continuing to investigate this issue.
- resolved Jul 12, 2020, 09:13 PM UTC
This incident has been resolved.