Fremont, CA, US - Investigating single physical server issues

Update timeline

1. investigating Oct 17, 2024, 05:24 AM UTC

   Our technical team is investigating issues on a single physical server impacting a small number of clients at the moment. Once we have a cause identified we will update this incident with more details.

2. identified Oct 17, 2024, 05:32 AM UTC

   This system is having a storage level issue impacting the availability of client VPSs. Staff are investigating at this time if this is a recoverable issue, or if its a hardware failure or data corruption. If needed clients will be restored to the most recent full disk off server snapshots available. We will update this incident with further details as soon as possible. Thank you for your patience as our team works on this issue.

3. monitoring Oct 17, 2024, 05:53 AM UTC

   We have unfortunately identified that it a hardware failure that caused loss of data for the affected systems. Steps we are taking: 1) Impacted VPS systems are being restored from our weekly off-server/off-site full snapshots of each VPS systems hard drive. This will restore each client's VPS system to exactly how it was on Oct 9th. As this is the morning of the 17th which would have been the next backup window, unfortunately we have to restore back to the 9th. 2) Once this is complete and clients are back online. We will be reviewing which impacted clients also have purchased our add-on Advanced Daily File/Database backup solution, and we will start the recovery process of files/databases from those backups. 3) Staff will be contacting the email contact we have for each account in our Account Portal to inform them of the restoration that took place so if they have their own off-site backups they can restore from there if needed. Again thank you for your patience as our team works on this restoration process.

4. monitoring Oct 17, 2024, 06:23 AM UTC

   Just crossed the 25% restoration mark. We will update again in just a bit.

5. monitoring Oct 17, 2024, 06:45 AM UTC

   At the 50% restored mark.

6. monitoring Oct 17, 2024, 07:15 AM UTC

   75% of impacted clients are now restored.

7. monitoring Oct 17, 2024, 08:03 AM UTC

   At about 90% restoration, unfortunately restoration speed has to do a lot with how much data is being recovered for each client and some of these are very large. Hoping the rest will be completed in about the next 30 minutes.

8. resolved Oct 17, 2024, 08:49 AM UTC

   All impacted client systems have been restored, and staff will be emailing each impacted client to make sure each knows of the restore that took place. Please feel free to always reach out to our support team at [email protected] if you have any questions.