Yandex Cloud experienced a minor incident on November 14, 2020 affecting Virtual Private Cloud and ru-central1-a and 1 more component, lasting 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 23, 2020, 10:01 AM UTC
On November 14, at 10:00 (GMT+3) we noticed that the load on DNS has increased, and some users may had drops and increased delays.
- resolved Nov 23, 2020, 02:36 PM UTC
The issue was solved on November, 14, at 15:00 (GMT +3). We will provide details later.
- postmortem Dec 11, 2020, 10:28 AM UTC
Incident summary On November 26, from 20:15 to 21:02 Moscow time, for some virtual machines in the b availability zone, the cloud DNS response time increased to several seconds. 20:15 - we've done some works to enable mirroring of DNS traffic to new DNS servers in the b-AZ and changed the cluster configuration. 20:24 - tests with DNS-resolver problems lit up, and our on-duty engineer started working. 20:37 - a rollback of mirroring DNS traffic in the b-AZ started. By this time, some of the cluster servers have received a new configuration. 21:02 - Disabling mirroring has come to all servers; the cloud DNS response returned to average values. What happened To prevent problems with DNS (https://status.cloud.yandex.ru/incidents/229) and to prepare for Black Friday, we introduced additional DNS resolvers on the platform. New DNS resolvers worked for a long time in the a-AZ and about a week in the c-AZ in shadow-mode (their responses are ignored, but they can be switched to operate at any time). It was supposed to add new resolvers to all zones with an additional timeout for responding so that in case of problems with the primary DNS, you could switch from shadow-mode. A timeout is needed so that the new DNS servers act as spare only if response delays happen in the primary DNS. In cloud DNS, each DNS request is sent to the resolver servers, and then a quorum of a certain number of responses is collected, and the best one is transmitted to the virtual machine. This implementation can be explained by the fact that resolvers are configured at different speeds and may respond in different ways when changed. There's been a mistake in the configuration during the works. The service that chooses the best response began to expect it not from some resolvers, but all. The new DNS resolvers were configured with an additional timeout, while the main ones periodically performed a full configuration update, during which they ignored the incoming requests. These factors resulted in a response time of up to several seconds, which exceeds the typical timeout for client applications and libraries. Measures to prevent a similar situation from recurring in the future • The configuration of the service that proxies DNS requests is in the correct state in the b-AZ. • Full migration to new DNS resolvers would simplify configuration, significantly reducing the risk of errors.