phoenixNAP experienced a major incident on October 12, 2024 affecting Data Security Cloud (DSC) and Backup as a Service (BaaS) - Veeam Managed Backups and 1 more component, lasting 8h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 12, 2024, 12:06 PM UTC
Investigating - We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are actively troubleshooting the issue. Client Impact - Virtual machines may be experiencing high read and write latency. If you have any questions about this incident, feel free to reach out to [email protected].
- investigating Oct 12, 2024, 04:46 PM UTC
We are currently experiencing an issue with our PHX DSC VCD environment. This issue has been isolated to a backing storage array. We have engaged the vendor and are actively troubleshooting the issue. Client Impact - Virtual machines may be experiencing high read and write latency. If you have any questions about this incident, feel free to reach out to [email protected].
- identified Oct 12, 2024, 06:03 PM UTC
We are continuing to work with the storage vendor to resolve this issue. At this time, the storage vendor has further isolated this issue to a software bug and is currently working apply a potential fix to the impacted storage array. Client Impact - Virtual machines may be experiencing high read and write latency. If you have any questions about this incident, feel free to reach out to [email protected].
- identified Oct 12, 2024, 07:29 PM UTC
Currently, the vendor is continuing to apply the fix update. We have started to see the system normalize over the last hour as the fix update is being applied. However, please keep in mind that there is a chance that the latency will spike again during the update process due to the work being completed to resolve this issue. If you have any questions about this incident, feel free to reach out to [email protected].
- resolved Oct 12, 2024, 08:48 PM UTC
Hi there, We have identified the issue that was causing disruptions on DSC VCD environment in Phoenix and have completed implementing the fix. We are now seeing latency subside and throughput is at the expected levels again. We are continuing to monitor the situation to ensure further issues do not arise. For any questions or concerns or if you are still experiencing issues, please contact us at [email protected]. phoenixNAP NOC / phoenixNAP Cloud Services