Servers Australia incident
Intermittent Service Degradation for vCloud Director User Interface
Servers Australia experienced a minor incident on February 3, 2025 affecting Virtual Data Centre (VDC), lasting 14d 17h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 03, 2025, 04:22 AM UTC
We are currently investigating an intermittent issue with vCloud Director that causes some operations within the user interface to fail. We will provide updates as they become available.
- investigating Feb 03, 2025, 05:24 AM UTC
As part of the normal troubleshooting procedure, we need to restart the vCenter Management Server. There will be no customer workload impact, but the vCenter UI will be unavailable for approximately 10 minutes. This work is being performed at 16:25 AEDT.
- monitoring Feb 03, 2025, 05:57 AM UTC
After a reboot of the VCS environment, operations have returned to normal. We are monitoring the services and will update as needed.
- investigating Feb 03, 2025, 01:49 PM UTC
We are currently investigating an intermittent issue with vCloud Director that causes some operations within the vCloud Director user interface to fail. We need to restart the vCenter Management Server and the vCloud Director Server, There will be no customer workload impact, but the VCS and VCD User Interfaces will be unavailable for 15 minutes. This work is commencing now.
- monitoring Feb 03, 2025, 02:04 PM UTC
After a reboot of the VCD environment, operations have returned to normal. We are monitoring the services and will update as needed.
- monitoring Feb 03, 2025, 10:57 PM UTC
We are once again investigating an issue with vCloud Director that causes some operations within the user interface to fail. We will provide updates as they become available.
- identified Feb 04, 2025, 05:12 AM UTC
We have identified an issue with the hardware hosting the vCenter appliance. The appliance is being moved to a new host and will be unavailable for approximately 10 minutes. This will only impact the availability of the vCenter WebUI and not customer workloads.
- monitoring Feb 04, 2025, 05:58 AM UTC
Our Systems Engineers have rectified issues that were causing certain operations to fail within the VCD platform. We have tested functionality and can no longer replicate previous issues seen. Our Engineers will continue to monitor the VCD platform throughout the night for any irregularities. Please reach out to our support team if you encounter any issues with your VDC or Private Cloud environment.
- identified Feb 04, 2025, 01:30 PM UTC
Engineers have found that certain API calls to the backend from the vCloud Director interface are timing out. Emergency maintenance will start shortly and the vCloud Director user interface will be unavailable during this time. Estimated time is 30 minutes.
- monitoring Feb 04, 2025, 02:50 PM UTC
Maintenance is complete and all services are back online. We will continue monitoring and will update this status further if there are any more changes.
- investigating Feb 04, 2025, 10:50 PM UTC
We are once again investigating an issue with vCloud Director that causes some operations within the user interface to fail. Our Systems Engineering and Networking teams are actively investigating this issue in collaboration with the software vendor and their support teams, maintaining constant communication throughout the process. We will provide updates as they become available.
- identified Feb 05, 2025, 04:39 PM UTC
The VCD user interface will be temporarily unavailable while we perform emergency maintenance and updates. Engineers will update this status once complete.
- investigating Feb 05, 2025, 05:59 PM UTC
The VCD platform is back online. We are continuing to work with the software vendor to resolve intermittent issues in the UI. We will keep this status updated as work progresses and with further maintenance.
- investigating Feb 10, 2025, 03:31 AM UTC
We are continuing to work with the Vendor on this issue. As part of the process, we need to export a log bundle from the vCloud Director server. Doing this may cause the VDC interface to become unresponsive while the log is generated. There will be no impact to customer workloads while this takes place.
- monitoring Feb 10, 2025, 10:32 PM UTC
Servers Australia Engineers have identified that specific database queries executed by the vCloud Director system during inventory collection were causing the sync status between vCloud director and vCenter to enter an invalid state, causing the vCenter to continually try to resync and never complete. This caused specific objects from vCenter to be unable to be collected into the VCD database, as well as affecting creating of new VMs while the sync status was not healthy. This issue was traced back to a bug in the error handling code within the vCloud Director application itself, which made the root cause difficult to pinpoint. To address this, we have implemented changes to mitigate the issue and restore functionality. We will continue to monitor the situation closely to ensure the stability of the system before formally closing the status. We appreciate your patience and will provide further updates as needed. Thank you, SAU Engineering Team
- resolved Feb 17, 2025, 09:54 PM UTC
This incident has been resolved.