Anchor Host experienced a critical incident on August 11, 2020 affecting Hosting, lasting 13h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 11, 2020, 06:30 PM UTC
Some customers with websites located within the Northern Virginia (US East 4) datacenter are currently offline. This appears to be an issue with a particular Google datacenter. Will report back as new information is known.
- identified Aug 11, 2020, 06:41 PM UTC
This issue has been narrowed down to an individual LXD within the Northern Virginia (US East 4) datacenter. Kinsta is aware of the issue and is working to restore. https://status.kinsta.com/incidents/qc639q78n6fg
- identified Aug 11, 2020, 08:35 PM UTC
Still no ETA on when affected websites will be restored. Kinsta is working with Google Support which is taking longer then usual for this type of issue. This outage is isolated to a individual hardware failure within a Google datacenter.
- identified Aug 12, 2020, 01:54 AM UTC
Update from Kinsta: The ongoing issue is due to an apparent hardware failure at Google Cloud. We've been working with Google Cloud Support seeking resolution but at this time an ETA on restoration of the affected virtual machine is unavailable from Google Cloud. Due to the ongoing uncertainty regarding when the hardware failure will be resolved, we've made the decision to restore the affected machine from our most recent backup. Backup restoration is in process now. We will continue working with Google to seek the retrieval of all data leading up to the point where the hardware failure occurred and we will provide additional information as soon as possible.
- monitoring Aug 12, 2020, 07:11 AM UTC
From Kinsta, At approximately 2:20 PM EST, 11 August 2020, one of our Google Cloud virtual machines went offline due to an apparent hardware failure. We worked on this issue with Google Cloud Support for multiple hours without success. Eventually, we determined that our best course of action was to restore the affected sites using our most recent backup. We create backups of all of our virtual machines every four hours. In this case, the most recent backup was approximately three hours old at the time that the incident began. As a result, the backup we have restored does not contain any changes made to affected sites after 11:07 AM EST, 11 August 2020. As of 11:22 PM EST 11 August 2020, all LIVE sites which were affected by this incident have been restored from that backup. Our Engineering team is continuing to work on restoring affected STAGING sites. At this time, we are focused on the recovery of website data between the 11:07 AM and 2:20 PM EST. We will keep you updated on our progress as it pertains to recovering this data. We are deeply disappointed by this extended outage. Thank you for your patience with us during this event. We will continue to keep you updated regarding this incident until we are confident it has been fully resolved. Thank you for being a valued Kinsta customer.
- resolved Aug 12, 2020, 07:53 AM UTC
All Live and Staging sites have been restored. The services are all fully available now.