Emergency Maintenance - Physical Server Reboot - Corrective Action

Update timeline

1. scheduled Aug 02, 2014, 08:28 PM UTC

   We have been alerted to an issue with the RAID 1 boot controller on a single physical server in our VMware Clusters, this maintenance window will only affect a small subset of our clients, not all clients will be affected. This issue affects the management and monitoring of this specific physical server. While this issue currently is not affecting client's virtual machines, it impacts our ability to move clients off this physical server or make resource changes to client's running virtual machines currently powered by this physical server. To correct the issue we will be doing a reboot of the one physical server tonight. This will impact clients with virtual machines on this specific physical server, no other clients will be affected. In the event we find deeper issues we may need to do other work to correct the issue, so we are going to put a maintenance window size of 2 hours on this incident for now. Staff will update this announcement as the emergency maintenance progresses tonight. Shortly our Support Team will also be emailing each of the clients contact email address that we know will be affected by this window, referring them to this announcement. If you have any questions at all, please contact our Support Team. Thank you. Nevin Lyne - Director of Technology

2. scheduled Aug 03, 2014, 01:59 AM UTC

   Maintenance will begin as scheduled in 60 minutes.

3. in progress Aug 03, 2014, 03:00 AM UTC

   Scheduled maintenance is currently in progress. We will provide updates as necessary.

4. in progress Aug 03, 2014, 03:24 AM UTC

   Ok, a simple reboot of the physical box did not correct things, we are investigating further and will update this incident as soon as we have a bit more informaiton.

5. in progress Aug 03, 2014, 04:01 AM UTC

   Ok looks like the raid controller took out the VMware boot volume. This is where VMware boots from and not where user data is stored. We are working on replacing the raid controller and re-installing the base VMware hypervisor so we can get virtual machines back online as soon as possible.

6. in progress Aug 03, 2014, 04:55 AM UTC

   Sorry for the delay in an update. We are very close to getting client VMs powered back up on this physical host. Staff are working as fast as possible while making sure client data is protected. Thank you.

7. completed Aug 03, 2014, 05:15 AM UTC

   2014-08-03 12:17am - The scheduled maintenance window closed but we are still working on the final steps to power up client virtual machines. We will be updating this entry still as we move forward in this emergency maintenance. 2014-08-03 1:13am - Many clients are back online, we are still working as quickly as possible to get the rest of the affected VMs fully back online.