phoenixNAP experienced a minor incident on October 1, 2024 affecting Data Security Cloud (DSC), lasting 1d 19h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 01, 2024, 09:44 PM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 01, 2024, 11:31 PM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 12:02 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 12:31 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 01:02 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 01:33 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 02:00 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 02:34 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 03:00 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 03:30 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 04:06 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 04:30 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 05:00 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 07:50 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 10:42 AM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. There is currently no ETA on the hardware arrival. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- identified Oct 02, 2024, 11:14 AM UTC
Update - We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are pending replacement hardware which is being shipped as swiftly as possible. Hardware delivery ETA 9:45 AM - 1:45 PM MST. The phoenixNAP Cloud team is continuing to troubleshoot and investigate a solution to help alleviate the experienced latency and/or extended time it is taking to complete VM management functions such as powering on/off, migrations, etc. Client Impact - Virtual machines may be experiencing latency and/or extended time completing VM management functions such as powering on/off.
- monitoring Oct 03, 2024, 03:56 AM UTC
The drive was delivered and has been replaced with the help of the HW team. The storage array is now showing as healthy with minimal latency. We will continue to monitor to ensure no further issues arise. Client impact – None
- resolved Oct 03, 2024, 05:26 PM UTC
After monitoring, our engineers were able to confirm this issue has been resolved. Should you experience any further issues, please contact our support team at [email protected] Client impact – None