Yandex Cloud experienced a major incident on February 5, 2020 affecting Compute Cloud and Container Registry and 1 more component, lasting 2h 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 05, 2020, 03:13 PM UTC
Starting from 16:05 (GMT +3) we are facing issues in Yandex network. Users may face partial or complete services outage in Yandex.Cloud. We are working on this issue and will provide an update additionally.
- resolved Feb 05, 2020, 03:45 PM UTC
Issue was solved around 18:35 GMT+3. We will prepare and publish detailed report about this incident after 10.02.2020.
- postmortem Feb 06, 2020, 07:46 AM UTC
On February 5 from 16:05 to 18:35 (Moscow time), Yandex.Cloud users experienced problems with an access to Yandex.Cloud services via the management console and API. We apologize to all users affected by this incident and want to tell you more about what happened and what measures are being taken to prevent similar situations in the future. What happened?What happened? On February 5, the internal DNS service failed during the regular maintenance works aimed to improve the fault tolerance of Yandex services. As a result, some of the company's services, including Yandex.Cloud, were unavailable. ReasonsReasons Yandex performs regular maintenance works several times a month to increase the fault tolerance of the services. During these works, we disable one of the data centers to make sure that all services can process all user requests under the maximum load without a services’ degradation. The works are organized in a way that doesn’t affect the proper operation of Yandex Services, including Yandex.Cloud, and doesn’t affect the resources and data allocated in the cloud platform by our clients. The data center is disabled for all services in several stages: • The services move the workloads to other data centers in a soft mode. • L3 load balancers stop sending external traffic to the disabled data center. • We make sure this doesn't degrade the performance of our systems and services. • If the check is successful, we disable network connection between the data center, where the maintenance works are being performed, and the rest of the Yandex network. On February 5, we started regular works according to the procedure above: • 16:00 — We successfully redirected the workloads to other data centers. • 16:03 — L3 load balancers isolated the chosen data center from external traffic. • 16:04 — We noticed DNS service performance issues in the remaining data centers. The internal Yandex DNS service is organized as follows: • To ensure fault tolerance, we use several DNS cache farms in different data centers. • On the list of DNS resolvers on the servers, the local cache goes first, followed by the DNS cache farms. When L3 load balancers stop routing external traffic to the disabled data center, its servers, with network connectivity maintained, access the DNS cache in other data centers. • Due to its architectural features, the DNS cache of the active data centers doesn't have any information about the required records that are stored in the caches of the disabled data center. Therefore, it starts actively accessing the farms of authoritative servers. These servers store all the information about the address zone they are responsible for. On February 5, the load was higher during the maintenance works, and the farms failed: the response time and the number of repeat requests increased. The effect caused by the sharp increase in the number of requests to the DNS caches was intensified as one of the internal services by mistake didn’t lower and turn off its load. This finally caused the authoritative servers to go down. To quickly fix these problems, we added authoritative DNS servers to the farm and disabled internal services in the chosen data center by 17:45. By 18.35, we managed to stabilize the situation completely. Measures to prevent this situation in the future:Measures to prevent this situation in the future: • We are increasing the number of DNS servers that serve internal and external requests. • We're changing the architecture of the DNS caching system to enhance its performance. • We're dividing the authoritative server DNS farms for external and internal users and are increasing their performance. For internal high-load services, dedicated DNS farms will be added. • All Yandex.Cloud address zones will have their own independent authoritative DNS server farms.