Yandex Cloud experienced a major incident on June 8, 2022 affecting Cloud DNS and ru-central1-a and 1 more component, lasting 1h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 08, 2022, 06:50 PM UTC
We noticed some problems with the DNS. The developer team has been notified and currently investigating this issue.
- resolved Jun 08, 2022, 07:33 PM UTC
Issues were solved at 22:29 (GMT +3). All Cloud services are working correctly. We're investigating the causes of the problem.
- postmortem Jul 06, 2022, 03:04 PM UTC
On June 8, during the period from 21:00 to 21:57 Moscow time, users experienced difficulties with service access due to NS servers overload. We apologize to all users affected by the incident and want to share more details about what happened and prevent similar situations in the future. What happened?What happened? Yandex Cloud has several authoritative NS servers processing clients' requests. For example, if a client wants to find out which IP address is assigned to cloud.yandex.ru, the following happens: • The request goes to the root server which returns a ru-Zone NS server • Request is being redirected to that server and it replies with locations of .yandex.ru zone NS server • NS server responsible for the yandex.ru zone knows what cloud.yandex.ru NS servers are. They, in turn, reply with a cloud.yandex.ru record with a particular IP address Due to maintenance, we performed zone relocation from one authoritative NS-server to another. All similar works we had before brought no problem. Chronology of the incident: • 16:00 - zones were relocated between internal authority NS servers as planned • 21:00 - NS servers became overloaded with requests • 21:57 - servers operation returned to normal As we know by now, NS servers' performance got degraded due to maintenance. This became the root cause of partial failure to respond to some DNS requests. On June 9 at 01:00 Moscow time, the incident was resolved and we rolled the changes back. What has been done to prevent a similar situation in the future:What has been done to prevent a similar situation in the future: • We are constantly monitoring authoritative NS servers' performance capacity including organic traffic checks • We transferred authoritative NS servers to dedicated capacities