Yandex Cloud incident

VM availability issues

Minor Resolved View vendor source →

Yandex Cloud experienced a minor incident on February 26, 2021 affecting Compute Cloud and Managed Service for ClickHouse® and 1 more component, lasting 1h 55m. The incident has been resolved; the full update timeline is below.

Started
Feb 26, 2021, 10:05 AM UTC
Resolved
Feb 26, 2021, 12:00 PM UTC
Duration
1h 55m
Detected by Pingoru
Feb 26, 2021, 10:05 AM UTC

Affected components

Compute CloudManaged Service for ClickHouse®Managed Service for Kubernetes®Managed Service for MySQL®Managed Service for PostgreSQLManaged Service for Valkey™Yandex StoreDocru-central1-aru-central1-b

Update timeline

  1. investigating Feb 26, 2021, 11:14 AM UTC

    We noticed that some users have availability VMs issues. The developers are already aware and are fixing them.

  2. resolved Feb 26, 2021, 12:20 PM UTC

    The issue was solved at 15:00 (GMT +3). We're investigating the causes of the problem.

  3. identified Mar 11, 2021, 09:38 AM UTC

    During the update to the latest version of OpenBMC for Yandex.Cloud hosts an error was made in the filter that determined a «canary» selection of hosts to update. Because of this a small group of hosts from the production cluster was included in the sample. During the update the hosts were rebooted, which resulted in an automatic migration of the virtual machines. We are changing the technological procedure for updating the BMC, which will limit the ability to select hosts for the production cluster in the future.