Yandex Cloud experienced a minor incident on May 25, 2021 affecting Virtual Private Cloud and ru-central1-b, lasting 2d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 25, 2021, 08:11 PM UTC
We have noticed network connectivity issues for a small part of users in zone C . The developers are already aware and are working on fixing them.
- identified May 26, 2021, 12:52 PM UTC
From 15.30 to 15.50(GMT +3), there was a spike in errors that affected the operation of services in zone ru-central-b and the web console.
- identified May 27, 2021, 12:06 PM UTC
Our engineers took measures to remove the negative effect, and at the moment the services are working correctly. We will close the problem as soon as we are convinced of the effectiveness of the applied solutions.
- resolved May 28, 2021, 11:14 AM UTC
After taking action on 27.05 and monitoring the state of network connectivity for a long time, there were no more errors. All Cloud services are working correctly. We will inform you about the reasons for the incident later.
- postmortem Jun 01, 2021, 03:14 PM UTC
Incident SummaryIncident Summary We were experiencing unstable operation of the network in the zones ru-central1-b, ru-central1-c on May 20 (21:25-21:36 MSK), on May 25 (13:34-13:38 MSK, 22:09-22:15 MSK, 22:20-22:30 MSK), and May 26 (15:40-15:48 MSK). The problem was resolved on May 27 at 15:06 and all services became normal functioning. After quarantine period, the incident was considered as finished. We apologize to all users affected by this incident, and would like to give more details about it and the measures we have taken to prevent a recurrence of such situations in the future. What happened?What happened? In the cloud, the network is built in such a way that its controllers exchange routing information with nodes that are responsible for external connectivity, and with daemons that are responsible for the overlay network on the physical server. On May 20 due to the increase in traffic volume, there was a major increase in the number of nodes responsible for external connectivity. This resulted in a noticeable increase in the amount of routing information, which led to the original issue on May 20 during the scaling itself. The controllers were overloaded and could not serve the daemons on the physical server, which led to the flushing of routing information on the physical servers. In addition, the nodes that are responsible for external connectivity still can detect inconsistencies in routing information and initiate its repair in case of problems. Unfortunately, the mechanism for fixing the inconsistency of routing information requires the exchange of this information between nodes, which on May 25 and 26 led to a repeating overload of network controllers, which again was accompanied by a partial loss of connectivity. Measures to prevent a recurrence of the situation in the futureMeasures to prevent a recurrence of the situation in the future • The mechanism for repairing the inconsistency of routing information is temporarily disabled and starts only under service's duty control. • The logic of the daemons providing the overlay network has been changed taking into account the possible overload of network controllers • Service developers are optimizing the routing information exchange scheme