Yandex Cloud incident

Problems with creating and starting virtual machines

Minor Resolved View vendor source →

Yandex Cloud experienced a minor incident on May 13, 2020 affecting Compute Cloud and Data Processing and 1 more component, lasting 2h 28m. The incident has been resolved; the full update timeline is below.

Started
May 13, 2020, 08:20 AM UTC
Resolved
May 13, 2020, 10:48 AM UTC
Duration
2h 28m
Detected by Pingoru
May 13, 2020, 08:20 AM UTC

Affected components

Compute CloudData ProcessingManaged Service for ClickHouse®Managed Service for Kubernetes®Managed Service for MySQL®Managed Service for PostgreSQLManaged Service for Valkey™Yandex StoreDocru-central1-aru-central1-b

Update timeline

  1. investigating May 13, 2020, 09:33 AM UTC

    Some users may experience problems when creating or starting virtual machines with 2 vcpu and 2Gb ram configurations. The developer team is notified and currently investigating this issue.

  2. identified May 13, 2020, 11:21 AM UTC

    We noticed some problems with the control plane in the Compute Cloud service. Some users may encounter errors while creating, stopping, and adjusting VMs and hosts of MDB clusters. The developer team is notified and currently investigating this issue.

  3. identified May 13, 2020, 11:52 AM UTC

    Issue was solved at 13:48 (GMT +3) We will provide details later.

  4. resolved May 14, 2020, 12:13 PM UTC

    After the operations of stopping, dynamic migration, and VM removal, it's necessary to perform the procedure of cleaning Compute resources. A particular internal service can solve this task – Garbage Collector. The request for a list of removed resources in the GC is written for less load. Until a certain point, it worked correctly, but on May 13, it ceased to fit into a timeout. It led to an actual halt of the Compute Cloud resource cleanup process, which is why they started to run out. The Cloud team is already working on a systematic solution to this problem.