Partial network unavailability

Update timeline

1. investigating Oct 16, 2024, 12:58 PM UTC

   We have noticed occasional short-term network connectivity problems with a number of resources in ru-central1-a. It is also possible that clusters of managed services are affected. The developers are already aware of the situation and are working on fixing it.

2. identified Oct 16, 2024, 01:07 PM UTC

   Network problems are observed on some new VMs in the ru-central1-a availability zone.

3. identified Oct 16, 2024, 04:55 PM UTC

   Developers have fixed the root cause of the issue at 19:30 (GMT+3). We have established full availability recovery at 19:45 (GMT+3) according to our monitoring systems. Currently we are working to stabilize the situation and to prevent the issue from reappearing.

4. resolved Oct 16, 2024, 06:49 PM UTC

   There were no regressions in 19:45-21:42 (GMT+3) interval, technical team now has finished the correcting measures.

5. identified Nov 02, 2024, 02:13 PM UTC

   Update on Work Results to Prevent Incident Recurrence. We have increased the resources for VPC controller clustering in all zones and implemented optimizations in the algorithms for processing and distributing routing information. This has accelerated the execution of extensive routing processing scripts by orders of magnitude. As a result, the risk of controller overloads, which led to the incident, has been significantly reduced, and the time needed to deploy new controllers in case of existing ones overloading has been reduced substantially as well. At this point, the primary operational work on the VPC service to prevent recurrence has been completed, greatly reducing its likelihood. However, we continue to work on long-term efforts aimed at further reducing the risk. We plan to provide a detailed update on these plans in December.

6. postmortem Nov 05, 2024, 12:39 PM UTC

   Short description. On 16th of October, the VPC service team conducted updates on VPC controllers in the ru-central1-b availability zone (hereinafter referred to as the AZ), responsible for propagation routing information. After maintenance works, we discovered that the similar components in the ru-central1-a AZ transitioned into an unstable state that led to the significant degradation of routing information propagation across compute nodes running virtual machines (VMs). The lack of routing info resulted in network unavailability for newly created VMs, while VMs already running at the time remained accessible. Later on, during the investigation on our side, an unexpected situation occured, causing partial unavailability for approximetely one-fifth of all VMs in ru-central1-a. Below we will report on how this incident affected services, how we were able to resolve it, and the steps we are taking to prevent the risk of recurrence in the future. Impact on Yandex Cloud services (Timezone MSK+0 / GMT+3): • Compute Cloud: A temporary increase in task and operation processing times in ru-central1-a. • Virtual Private Cloud: Lack of network connectivity for newly created VMs, inability to open new network connections for about 20% of existing VMs in ru-central1-a. • Managed Service for Kubernetes: Failure of several dozen zonal clusters in ru-central1-a. Creating zonal clusters with masters in this AZ required manual reboots afterwards (after the incident). • Application Load Balancer: The service operated normally, requests from other AZs to backends in ru-central1-a could fail if a certain balancer would not have healthcheck rules (configured on user-side). • DataLens: Noticeable perfomance degradation, a significant portion of requests failed to complete between 17:15 and 17:30. • SpeechKit: An increase in timings was observed between 16:45 and 19:45, and some requests failed. The accumulated queue of recognition tasks had been in processing up to 23:30. • Translate, Vision OCR, SpeechSense, Foundation Models, YandexGPT API: An increase in timings was observed between 16:45 and 19:45, and some requests failed. • Serverless (e.g. API Gateway, Cloud Functions, Serverless Containers): From the start of the incident, requests routed to ru-central1-a experienced slightly increased timings. As a precaution, the operational components in this AZ were closed at 16:50 to exclude the risk of disruption. • Yandex Query: It was not possible to execute queries in several folders between 17:25 and 18:26. • Managed Databases (core): It was impossible to create or modify clusters with masters in ru-central1-a. • Yandex MetaData Hub: Errors in creating and updating connections were observed between 18:22 and 18:40. Password retrieval did not work in all the clients. • Console: The service remained operational, but functions related to affected services returned errors. • Managed Service for PostgreSQL, Managed Service for MongoDB, Managed Service for Redis™: Clusters with no H/A (high-availability) configuration experienced perfomance degradation, up to complete unavailability. • Managed Service for Apache Airflow™: Several create/stop operations failed to complete successfully. • Yandex Data Proc: Impossibility to create clusters or submit jobs. • Managed Service for MySQL: Several hosts in ru-central1-a went into an inconsistent state and required service engineers' intervention to be restored. • Managed Service for Greenplum®: Clusters running in ru-central1-a became inaccessible for reading/writing. • DataSphere: It was not possible to start new notebooks, jobs or nodes in ru-central1-a. Existing active resources continued to operate. Projects in other zones were unaffected. Timeline October 16th, 2024 (Timezone MSK+0 / GMT+3): 12:00 - Maintenance starts for the control-plane in ru-central1-b. 15:03 - Restart of one of the controllers in the group due to the release pipeline. 15:09 - Transition of the control-plane in ru-central1-a (hereafter, all actions/events refer to this AZ) to an unstable state. Updates on routing information partially stops propagating within AZ. The service team begins investigation. The moment of an actual impact: newly created instances are unavailable, existing ones continue to operate normally. 15:15 - The service team registers an internal incident. 15:58 - The internal investigation confirms the impact potentially exceeding the service's SLA. The incident is posted on the public status board. 16:30 - The service team decides to reboot one instance of the controller group. Before that, a governed operation to remove the instance from the controller group was performed. 16:44 - One controller is rebooted. 16:50 - The service team finds out that the rebooted controller cannot establish peerings with existing controllers, as they do not respond to sessions' establishment attempts. 17:06 - It becomes clear that the entire group needs to be rebooted. A new group of at least three new controllers is created to transfer load. 17:15 - During the load transfer, an abnormal situation occurs, resulting in routing info loss on compute nodes. This leads to significant connectivity degradation for some existing VMs (about 20% of AZ). 17:30 - New controllers are transitioned into operation. 18:10 - Restoration of routing information. Degradation of connectivity is mitigated. 18:40 - Remaining old controllers are decommissioned and rebooted. 19:30 - The impact on newly created instanсes is removed. There're still isolated issues with some virtual agent controllers (on compute nodes) which do not directly lead to the very same impact. What happened, and what measures are we taking? General context of what's happening All recent incidents of the VPC service are correlated with the current state of controllers in the ru-central1-a AZ. Essentially, the control-plane subsystem of VPC is a large regional service of Yandex Cloud that performs several key tasks: • Exchanging routing information with virtual router controllers (vRouter-agent) running on each physical server in AZ (compute node); • Exchanging routing information with border virtual routers in AZ, providing external connectivity (IPv4, IPv6, NLB, NAT, CIC, PL); • Exchanging routing information with control-plane instances in AZ to ensure cross-AZ connectivity – effectively making the service regional; • Other important functions, including the announcement of static routes. Recently, we've observed periodic cases of the service degradation for control-plane in zone ru-central1-a in certain specific scenarios. This occurred during incidents on October 10, October 11, and October 21, impacting newly created VMs. We do understand the criticality of this service and the steadily growing load on it. Therefore, we consistently invest team effort into its development and the correction of systemic issues. Volume of routing information in ru-central1-a. What has been done since then? • Separated the delivery of configuration and routing information across different controller groups to parallelize different types of load; • Significantly optimized the pipeline for static routes propagation; • Implemented a data-plane operating mode where routing information is retained during control-plane failures, allowing for connectivity preservation between VMs in the event of correlated control-plane failures; • Supported protective mechanisms for offloading and returning load to the control-plane for planned and unplanned controller group changes; • Added resources to the control-plane group for rapid mitigation of potential adverse situations. What is planned to prevent future issues? • Over the next two weeks, we plan to add resources to control-plane groups in other AZs; • In Q4'2024, we are implementing a caching layer between the control-plane controllers and the virtual router controllers. • We have identified several optimizations that can improve efficiency in some complex scenarios. Development and testing are in progress, and based on the results, implementation is scheduled for the end of 2024 to the beginning of 2025. If you have any questions regarding the incident or need assistance, please contact Yandex Cloud Support through the Support Center or via email at [email protected]