Problems with launching new VMs in ru-central1-c zone, running VMs are not affected

Update timeline

1. investigating Sep 26, 2022, 07:22 AM UTC

   We noticed problems with running virtual machines in the ru-central1-c zone, running virtual machines are not affected. The developers are already aware and are fixing the problem.

2. identified Sep 26, 2022, 02:30 PM UTC

   Since 16:00(UTC+3), most operations have been completed. Single problematic operations remain. Our engineers are working on a complete elimination of the incident.

3. resolved Sep 26, 2022, 08:03 PM UTC

   The problem was fixed at 22:43 (GMT +3), currently the Cloud services are working correctly. We are investigating the causes of the issue.

4. postmortem Oct 11, 2022, 04:09 PM UTC

   09:00 MSK - the internal alerts for problems with creating and updating VPC objects have triggered, the engineers on duty have started to look for the cause. 14:00 MSK - found problems with control-plane VPC DB, started recovery. 16:00 MSK - the major part of pending VM start operations were finished. 22:43 MSK - all operations were finished, the problem was fully mitigated. What happenedWhat happened Because of incorrect manual addition of control-plane VPC node, the internal DB containing the information about virtual NICs and related networks was desynced. This forced the DB into Read-Only state, so some of the VMs in the C zone were unable to start. These VMs got stuck in the Provisioning state. Additionally, some dependant operations in managed services got stuck as well. Plans to prevent this from happening in the futurePlans to prevent this from happening in the future • We are going to make Add/remove control-plane VPC nodes procedure automatic - to exclude human factor in this part. • We've already added DB desync monitoring - this would help to find the cause faster. • We also decided to improve our internal tools helping us mitigate similar incidents to expedite the overall recovery process.