DNS resolving issues at ru-central1-a

Update timeline

1. investigating Apr 07, 2020, 06:28 PM UTC

   We noticed new problems with DNS resolving at ru-central1-a. Some users may notice service degradation. Developer team is notified and currently investigating this issue.

2. resolved Apr 07, 2020, 06:38 PM UTC

   Issue was solved at 21:29 (GMT +3). Please see detailed report below.

3. postmortem Apr 15, 2020, 06:44 PM UTC

   On April 7, from 13:41 to 14:32 (GMT+3) in the ru-central1-b availability zone, and from 21:03 to 21:29 (GMT+3) in the ru-central1-a availability zone, some of Yandex.Cloud users were facing issues with network accessibility on their virtual machines (VMs). During this malfunction, resolving of domain names of VMs in availability zones was working unstable, some of the DNS-server queries ended with SERVFAIL or REFUSED responses. We apologize to all users affected by this incident and want to tell you more about what happened and what measures are being taken to prevent similar situations in the future. What happened?What happened? Yandex.Cloud DNS-service consists of two parts. We call them the Data plane and the Control plane. The Data plane is responsible for resolving domain names; it gets DNS-queries from users and returns responses. The control plane tracks the information about users' clouds (deleting of VMs) and changes the configuration of the Data plane service. The Data plane service single-handedly resolves internal cloud addresses (for instance, VM and database addresses), sending the rest of queries on the upstream DNS-servers of Yandex. The failure of the Data plane caused the malfunction. ReasonsReasons We prepared a fix for the problem with the Control plane to increase the number of internal query handlers during the recursive resolving in the Data plane that triggered the incident on April 6. The release that contained the increase of this limit by ten times rolled out in the ru-central1-a zone on April 6, and everything went smoothly. In a few hours, we noticed another approach to the limit, but only on one of the servers in this zone; the rest showed a margin of at least 100 times. We decided not to block the rollout in other availability zones, but to deal with this anomaly later. The load on all servers in the cluster is the same, and the fact that one of them suddenly began to use a hundred times more query handlers more likely indicated a problem in the server rather than a problem in the release. On April 7, this release went into the ru-central1-b zone. After several hours of operation, at 13:09, the problem was reproduced on one of the servers, and in this zone. On-duty engineers began to carefully study what exactly distinguishes the problematic server from the rest. At 13:41, our monitors reported that the new increased limit was exhausted on all servers. It was odd, and the first thing we suspected was an abnormal load on our DNS-servers. The engineers began to search for confirmation of this hypothesis, but couldn't find anything because there was no abnormal load. Yet, the DNS-servers reported that the limit of query handlers was exhausted. At 13:55, the engineers launched a rollback of the release on one server in the ru-central1-b zone. It helped, and the rollback began on the rest of the servers. When everything stabilized in ru-central1-b zone, we rolled back the release from ru-central1-a zone. However, we didn't restart the Data plane services and assumed there should be no problems due to the less load and work in this zone during the day. Unfortunately, this was a false assumption. At 21:03, the monitoring in the ru-central1-a zone similarly reported that the limit was exceeded on all servers in this zone. It led to the rejection of part of the queries and the SERVFAIL code response. The engineers gradually restarted services on all servers, and the situation returned to normal by 21:29. It is the amount of time needed for a planned restart of all DNS-servers in the cluster of one availability zone. As it turned out later, the cause of the limit problem was the inconsistency of the Data plane parameters. We increased the number of recursive query handlers and left the number of threads in which they were running unchanged, and it led to problems. Measures to prevent this situation in the future:Measures to prevent this situation in the future: • We rolled back the problematic release and fixed the detected inconsistency of the service parameters. • We conducted in-depth stress testing of the new release to understand its behavior under various load profiles better and will release it soon. • We'll conduct similar stress testing for all future changes that can potentially affect performance.