Yandex Cloud experienced a minor incident on April 6, 2020 affecting Virtual Private Cloud, lasting 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 06, 2020, 02:15 PM UTC
We noticed some problems with DNS in ru-central1-c availability zone. Developer team is notified and currently investigating this issue.
- resolved Apr 15, 2020, 05:23 PM UTC
Issue was solved at 17:40 (GMT+3). Please see detailed report below.
- postmortem Apr 15, 2020, 05:23 PM UTC
On April 6, from 16:51 to 17:40 (GMT+3) Yandex.Cloud users were facing issues with network accessibility on their virtual machines (VMs). During this malfunction, resolving of domain names of VMs in ru-central1-c availability zone wasn't working, and most of the DNS-server queries ended with SERVFAIL or REFUSED responses. We apologize to all users affected by this incident and want to tell you more about what happened and what measures are being taken to prevent similar situations in the future. What happened?What happened? Yandex.Cloud DNS-service consists of two parts. We call them the Data plane and the Control plane. The Data plane is responsible for resolving domain names; it gets DNS-queries from users and returns responses. The control plane tracks the information about users' clouds (deleting of VMs) and changes the configuration of the Data plane service. The Data plane service single-handedly resolves internal cloud addresses (for instance, VM and database addresses), sending the rest of queries on the upstream DNS-servers of Yandex. ReasonsReasons On every machine in the DNS-server cluster, both Data and Control plane services are raised. The servers' performance doesn't depend on each other, and the malfunction of either of them doesn't affect Yandex.Cloud DNS-resolving. The Control plane service generates configuration on its own, for its local service – the Data plane. On April 6, at 16:47:05, the Control plane service had an emergency crash and restarted on one of the servers. At the time of the launch, the Control plane service clears the configuration for the Data plane and then generates it from scratch. It takes up from 5 to 10 minutes. After the restart, it loaded all the necessary information into itself and proceeded to create configuration files for the Data plane service. At 16:51:05 and 16:51:15, the Control plane services also dropped and restarted on two additional servers. At this point, on-duty engineers began to make out why the Control plane services were falling. Unfortunately, the crashes continued. The services failed and restarted on all DNS servers in the ru-central1-c availability zone, and continued to restart about once per minute. The fall of one service caused a massive reconnection of virtual routers to its copy on another server, which also led to its fall. Thus, it formed a chain of cyclic crashes and restarts of the Control plane services on all servers. At the same time, the Data plane services continued to work, but since the Control plane cleared the configuration during their restarts, in reality, the Data plane did not do its job and responded to user requests with the REFUSED code. By 17:04, on-duty engineers managed to stop the cascade of service crashes and start the service firstly on one of them, and by 17:07 on all the others. In the next ten minutes, the newly raised Control plane services collected the necessary information about networks and VMs and prepared configuration files for the Data plane. At 17:16, Data plane services began to respond to part of the DNS requests correctly. Gradually, an increasing number of queries were processed successfully, but the full workability of DNS in ru-central1-c zone was restored only at 17:40. We are figuring out why this process took another 24 minutes. We were aware of the problem in the Control plane service, which led to crashes sometime before, and we developed a fix for it. This release managed to proceed to the zone ru-central1-a zone on April 6, and on April 7, it caused the next incident, which led to us rolling it back immediately. Instead, we took other actions to prevent the incident from occurring again and temporarily limited the connections of virtual routers to DNS-servers. Measures to prevent this situation in the future:Measures to prevent this situation in the future: • We blocked a possible reoccurrence of such malfunction and limited connections from virtual routers to the Control plane services and rolled back the problematic release for the Data plane. • We updated the Data plane's version, which does not clear the configuration in case of its potential crash and restarts. It will keep the Data plane operational even with a complete failure of the Control plane. • Shortly we will update the Data plane's version, that, during the restart, will immediately start working with the remaining configuration until the fall. After launching, the Control plane will correct the configuration in the areas where something has changed during the unavailability of the Data plane. • We will work to reduce the time of the Control + Data plane bundle's returning to its fully operational state after any crashes of the services.