Problems with external network connectivity in the ru-central1-a zone

Update timeline

1. investigating Nov 29, 2024, 12:55 PM UTC

   We noticed problems with external network connectivity in the ru-central1-a availability zone. The developer team is notified and currently investigating this issue.

2. identified Nov 29, 2024, 01:41 PM UTC

   The incident status will be updated at 17:00 Moscow time.

3. identified Nov 29, 2024, 02:21 PM UTC

   • There is a gradual recovery in the ru-central1-a zone - we expect a full recovery within an hour. Current impact: connectivity between cloud resources in this zone. • The situation is not expected to repeat in the ru-central1-b zone.

4. identified Nov 29, 2024, 03:20 PM UTC

   • Recovery continues in the ru-central1-a zone - we expect a full recovery within half an hour. Current impact: degradation of network connectivity between cloud resources in this zone. • In the ru-central1-b zone, there were periods of partial network unavailability of cloud resources in the period 16:40-16:53. Complete unavailability in the period 17:23-17:42. At the moment, no recurrence of the problems is expected. • In the ru-central1-d zone, there were periods of partial network unavailability of cloud resources in the period 17:12-17:29. At the moment, no recurrence of the problems is expected.

5. identified Nov 29, 2024, 04:00 PM UTC

   In the ru-central1-a zone, connectivity of cloud resources was fully restored at 18:25. Compute operations with network drives are slowed down in the ru-central1-a zone - recovery is expected within half an hour.

6. identified Nov 29, 2024, 05:06 PM UTC

   The Compute operation with network drives in the ru-central1-a zone remains slow - recovery is expected by 20:20.

7. resolved Nov 29, 2024, 05:35 PM UTC

   The execution of Compute operations with network drives in the ru-central1-a zone has returned to normal. All services are working normally. We consider the incident closed. If you still have questions about the operation of cloud resources, please contact technical support.

8. postmortem Dec 12, 2024, 06:33 AM UTC

   On November 29 an incident occurred in the ru-central1-a region. It was caused by inconsistencies in the routing table. A sudden increase in routing information flow led to the sequential failure of edge router clusters, stemming from memory exhaustion. This incident resulted in two major issues: 1. Loss of External Network Connectivity: Virtual machines in the ru-central1-a region experienced a disruption in external network connectivity. This included a failure of the Cloud Interconnect and Network Load Balancer, along with a partial failure of NAT. Additionally, this issue had a temporary partial impact on the ru-central1-b and ru-central1-d regions. 2. Loss of Internal Network Connectivity: There was also a loss of internal network connectivity within the ru-central1-a region via the internal Network Load Balancer. This led to operational delays in Compute Cloud, Managed Database, and Managed Kubernetes services. We apologize to all customers affected by this incident. In the following sections of this report, we will provide a detailed explanation of the causes of the incident and the steps we are taking to minimize the risk of its recurrence in the future. Root Cause Analysis and Detailed Incident Description The virtual network controllers have a built-in Graceful Restart mechanism designed to mitigate the impact of single controller failures. This feature enables the preservation of BGP peer routes, ensuring that routing information remains intact during cascading failures and thereby maintaining continuous traffic flow. However, it may lead to routing inconsistencies in individual virtual machines when transitioning between old and new routes. To address these discrepancies, the controller system incorporates corrective mechanisms that adjust and stabilize the routing data, guaranteeing seamless traffic flow to virtual machines. Following the incidents on October 16 and October 21, we optimized the route processing algorithm in the system for speed. Due to the fault-tolerant architecture of the chosen controllers, this optimization not only sped up normal operations but also accelerated the cyclical re-announcement of invalid routing information, which typically occurs during unexpected failures. On November 29, one of the network controllers failed. The combination of the aforementioned factors led to a situation where network controllers could disseminate several million route updates per second during peak times. Although the system's performance slowed down, it maintained its functionality. The edge routers (nodes responsible for external connectivity) were not prepared for such a high volume of routing information updates and could not decode the updates swiftly enough. This led to an increase in the update queue simultaneously across all instances of different external connectivity clusters, causing a corresponding increase in memory consumption. Consequently, the increased load on the edge routers led to their partial failure. Preventive Measures • Root Cause Addressed: The root cause of the controller failure, which triggered the incident, has been identified and fixed. • Enhanced Memory Management: The memory capacity was increased. Additionally, an extra memory exhaustion protection mechanism was implemented for edge router clusters responsible for external connectivity. • Simulation Testing: To verify the effectiveness of the implemented changes, a simulation of a similar incident was conducted in silico. • Route Update Management: The occurrence of cyclical updates of invalid routes was eliminated. Furthermore, extensive testing was performed to ensure that this issue does not recur in various failure scenarios. As of December 1, the root trigger and cause of this incident have been resolved. What's next Within a week, we will provide a report with further details on this incident and share our plans for enhancing network reliability. If you have any questions regarding the incident or require assistance, please contact Yandex Cloud Support through the Support Center or via email at [email protected].