Stable Host incident

vz01-ams.stablehost.com: Degraded performance

Major Resolved View vendor source →

Stable Host experienced a major incident on February 4, 2025 affecting VPS - Amsterdam, lasting 43d 23h. The incident has been resolved; the full update timeline is below.

Started
Feb 04, 2025, 08:39 PM UTC
Resolved
Mar 20, 2025, 08:24 PM UTC
Duration
43d 23h
Detected by Pingoru
Feb 04, 2025, 08:39 PM UTC

Affected components

VPS - Amsterdam

Update timeline

  1. identified Feb 04, 2025, 08:39 PM UTC

    There are two failed drives on the RAID array of this node. Some VPS may not start or work properly. We really suggest to keep your activity really low so the rebuild process can speed up. The first drive has been replaced and under rebuild process. Once the first one ends, the second failed drive will be replaced. Depending on the node activity, this process may take up to 12 hours. Your patience is appreciated during this process.

  2. identified Feb 05, 2025, 05:47 AM UTC

    The first disk rebuild finished. The second disk is being replaced however is taking way more time than expected. We will keep updating.

  3. identified Feb 05, 2025, 12:58 PM UTC

    The second drive rebuild failed, and they are looking over this to replace the disk and try the rebuild again. Our team at the datacenter have been working on this matter for the past several hours. The server remains in offline status due its delicate state. We will keep updating as long as we get any changes.

  4. identified Feb 05, 2025, 03:57 PM UTC

    The second failed RAID Drive is not rebuilding, therefore we are looking into more options to recover/restore the data. We will update the status as soon as we have more information.

  5. identified Feb 05, 2025, 04:57 PM UTC

    The data-center team is trying to replace the RAID controller and its cables. There is a chance this fix could rebuild the data.

  6. identified Feb 05, 2025, 10:59 PM UTC

    The RAID controller swap was made but without any luck. Therefore we are running a Bare Metal Restore with the latest backup we have. We will keep updated about the restoration progress.

  7. identified Feb 06, 2025, 02:27 AM UTC

    Restoration Progress: 30%

  8. identified Feb 06, 2025, 08:02 AM UTC

    Restoration Progress: 50%

  9. identified Feb 06, 2025, 12:06 PM UTC

    Restoration Progress: 70%

  10. identified Feb 06, 2025, 01:46 PM UTC

    Restoration Progress: 80%

  11. identified Feb 06, 2025, 03:26 PM UTC

    Restoration Progress: 90%

  12. identified Feb 06, 2025, 09:24 PM UTC

    Restoration has been concluded. We ran a filesystem check into the node before starting it. All VPS are now restored and running. We are reviewing and monitoring the server right now and we will give more updates in short.

  13. monitoring Feb 06, 2025, 09:24 PM UTC

    A fix has been implemented and we are monitoring the results.

  14. resolved Mar 20, 2025, 08:24 PM UTC

    This incident has been resolved.