Yandex Cloud experienced a major incident on October 28, 2021 affecting Network Load Balancer, lasting 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 28, 2021, 02:34 PM UTC
We noticed some problems with the loadbalancer service at 17:02 (GMT +3) . The developer team is notified and currently investigating this issue.
- resolved Oct 28, 2021, 03:14 PM UTC
The issue was solved at 17:25 (GMT +3). All Cloud services are working correctly. We're investigating the causes of the problem.
- postmortem Nov 17, 2021, 04:00 PM UTC
On October 28, from 17:03 to 17:14 Moscow time, network load balancers were unavailable in the ru-central1-c zone. We apologize to all users affected by this incident, and we want to tell you more about what happened and how to prevent similar situations in the future. What happened?What happened? There are two types of software components on load balancer hosts: • data plane - process network packets; • control plane - responsible for getting the balancer configuration, managing routing information, and balancing rules. A series of changes in the composition of the balancers, which took place on October 28 on the control plane, led to the fall of one of the control plane components on all load balancers. The automatic recovery mechanisms on each load balancer host classified this situation as abnormal and decided to restart the data plane and control plane components. As a result, the balancers in the ru-central1-c zone recalled the routing information announcements, and for several minutes going through a stop and start, loading all the necessary information. On October 30, we released a bug fix that caused the control plane to crash. Measures to prevent a similar situation from recurring in the futureMeasures to prevent a similar situation from recurring in the future • We revised the dependencies of the data plane and control plane components and the criteria for triggering the auto-recovery mechanisms - in the testing process. • Sharding load balancers in automatic mode - under development.