Yandex Cloud experienced a minor incident on January 21, 2021 affecting Virtual Private Cloud and ru-central1-a, lasting 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 21, 2021, 04:14 PM UTC
We noticed some problems with DNS In zone A. The developer team is notified and currently investigating this issue.
- resolved Jan 21, 2021, 04:23 PM UTC
We resolved the issue. DNS is in working conditions.
- postmortem Feb 09, 2021, 10:28 AM UTC
On January 21, from 18:48 to 19:15 Moscow time, all virtual machines in the availability zone-a stopped receiving responses from the cloud DNS. We made an error that led to their removal during our works on changing the DNS servers cluster's composition. At the time of the incident, the standby cluster was in an invalid state and was unable to replace the main cluster. What happened?What happened? In January, we began to put into operation a new version of the DNS service, with new features that will soon be available in Yandex.Cloud. To ensure the DNS servers' reliability, which serves the traffic, they were divided into two independent, simultaneously operating clusters. Clusters have different configurations; a timeout of at least one hour usually takes place between updates. On January 21, we carried out software updates on the backup cluster. We made an error in its configuration and, in addition to updating the code, the canary flag setting reset to its default value. The canary flag is used for testing. In this mode, the server is operated similarly to other servers, but the balancer ignores responses from it. By marking all servers in the cluster with this flag, the entire cluster's responses were ignored. Such responses ceased to reach the user. This problem went unnoticed, as the monitoring servers worked correctly, and the acceptance tests check both clusters at the same time. After some time, work began on the primary cluster to reduce the number of servers in operation. After the commissioning of the second cluster, some servers were no longer used. At this stage, a second error appeared: the exploited servers were automatically selected for removal. As a result, some servers were removed under the load, another started updating, and the backup responses were ignored. At this point, the servers stopped responding. We decided to roll back the applied configuration changes. Chronology of the incident:Chronology of the incident: • 16:59-17:18 — We carried out technical works to update the software on the backup cluster. The canary mode flag was added to the application configuration. • 18:48 — We started to work to reduce the number of servers in the main cluster. • 18:48-18:49 — The release engineer noticed the first signs of the problem: metrics stopped coming from the required servers. Tests with DNS server problems lit up, and our on-duty engineers joined in troubleshooting the problem. • 19:03 — We rolled back the main cluster configuration to the previous number of servers. • 19:15 — All the servers in operation are back in the process. Measures to prevent this situation from recurring in the future:Measures to prevent this situation from recurring in the future: • We will be run acceptance tests on each cluster individually. • The rules for updating the service have been tightened. We will update the clusters on different days, which will increase the likelihood of noticing unobvious errors. • Added to the schedule the procedure for the fastest rollback, which will reduce the recovery time by 10 minutes in a critical situation.