Yandex Cloud experienced a major incident on February 15, 2022 affecting Virtual Private Cloud and ru-central1-a and 1 more component, lasting 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 15, 2022, 11:02 AM UTC
We noticed some problems with connectivity with all zones . The developer team is notified and currently investigating this issue.
- resolved Feb 15, 2022, 11:06 AM UTC
Connectivity restored at 13:29 (GMT+3). We're investigating the causes of the problem.
- postmortem Feb 22, 2022, 12:26 PM UTC
On February 15, during the periods from 13:20 to 13:28 and from 13:51 to 13:53 Moscow time, we've observed the isolation of the ru-central1-b zone. Users perceived this as a loss of connectivity and unavailability of resources in the specified zone. We apologize to all users affected by the incident and want to share more details about what happened and prevent similar situations in the future. What happened?What happened? Availability zones are interconnected with two border routers that provide access to the backbone network. Both devices are by the same manufacturer but are controlled by different versions of same OS. The increase in signal load triggered a malfunction of the router's operating system, which led to a reboot of the interface module, and, as a result, to loss of connectivity and isolation of the availability zone. We restored the zone, contacted the manufacturer's technical support and took measures to prevent a similar problem in the future. Chronology of the incident: 13:19 - one of the edge devices terminated all BGP sessions with its neighbors. Traffic successfully switched to the second leg. Our monitoring recorded a short-term deterioration in connectivity. Services were not affected. During the analysis of the logs, we found a reboot of the router module responsible for managing interfaces, which caused a loss of communication with all connected devices and a drop in BGP sessions. 13:21 - the same thing happened with the device on the second route leg. By this time, the first device had not yet had time to recover, which caused the isolation of the ru-central1-b zone - the resources in it did not have access to the ru-central1-a, ru-central1-c zones, and the Internet. 13:24 - the components of the first device are back in operation. 13:26 - BGP sessions are up. The restoration of services has begun. 13:28 - according to the monitoring of the network and services, we've fully restored the connectivity. The technical support of the manufacturer participated in the analysis of the situation. 13:45 - we decided to reboot the first device. 13:47 - we applied commands removing the load from the shoulder. Traffic switches to the second. At the same time, the restart of the module on the second device happened again. This led to a reprised loss of connectivity and isolation of the availability zone. We've rolled back the changes and returned the load to the first shoulder. 13:53 - zone operation has been restored. 13:58 - we decided to reboot the second device where the problem recurred. 14:20 - the second device got back to work, and zone redundancy was restored. 15:00 - network operation has been stabilized, workload returned to the zone. Measures to prevent the recurrence of a similar situation in the futureMeasures to prevent the recurrence of a similar situation in the future At the moment: • We minimized signal load on routers. • We performed the organization of redundancy of failed equipment in all zones with a set of additional equipment in the Active-Standby mode. • We are testing a new version of router software in a lab environment. Long term: • Update the router software to a stable version after testing in a lab environment. • Increase the grouping of edge devices. • Add devices by another manufacturer.