Network connectivity issues in Zone A

Update timeline

1. investigating Jun 11, 2021, 12:46 PM UTC

   We have noticed network connectivity issues for some users in Zone A . The developers are already aware and are working on fixing them.

2. identified Jun 11, 2021, 01:29 PM UTC

   At 16.20 (GMT +3) the problem was fixed for all MDB services. Engineers are working to resolve the incident for the rest of the services.

3. resolved Jun 11, 2021, 01:42 PM UTC

   The issue was solved at 16:32 (GMT +3). All Cloud services are working correctly. We will provide details later.

4. identified Jun 11, 2021, 06:00 PM UTC

   We are working to avoid similar incidents in the future.

5. postmortem Jun 18, 2021, 07:24 AM UTC

   On June 11, from 15:14 to 15:19 Moscow time, we experienced network performance degradation, affecting at the peak up to 15% of virtual machines in zone A. In the period from 15:45 to 16:32, we observed DNS unavailability in zone A. It might have been looked like a lack of network connectivity at the application level. At the same time, in zones B and C, both the network and DNS were operational. From zones B and C, it was possible to access resources from zone A both by IPs and FQDNs. We apologize to all users affected by this incident. We want to tell you more about what happened and how to prevent a similar situation in the future. What happenedWhat happened Between May 20 and June 11, we observed an increase in the load on the cloud' network and balancers. We carried out several maintenance sessions to increase the capacity of network components by about 4-8 times. During maintenance work that was not supposed to affect the service availability, a software failure occurred in two of the five controllers of the overlay network, which is a part of the Virtual Private Cloud service (you can read more about how the service works here). For some users in Zone A, the outage might have been looked like a short-term loss of network connectivity. At the peak of the incident, up to 15% of the virtual machines in this zone were affected. A similar failure would in no way affect the operation of the network in zones B and C because they already use the updated logic of agents that ensure the ability of the overlay network in the event of a short-term failure of any number of controllers. Agents in Zone A were in the upgrade queue by the moment of the incident. Since the root cause of the problem was clear, we expected no significant impact on service health, so we tagged the incident as Minor on the platform component status board. But upon completion of the maintenance, it turned out that due to a software error in the BGP controller, which is responsible for advertising external IPv6 connectivity routes to the overlay network, one of the routes disappeared from the announcements. As incorrect routing information spread across the network, the cluster nodes began to lose IPv6 connectivity. DNS service, which uses IPv6 for internal communication, recieved incorrect information. From this point on, users could observe failures of FQDN resolving, which usually are necessary for applications to instantiate new connections. The already established IPv4 connections continued to work. TimelineTimeline 15:14 - two of the five overlay VPCs failed in Zone A. 15: 14–15: 19 - some users in Zone A have been experiencing problems with network connectivity. 15:30 - A software error in the BGP controller caused one of the routes to disappear from the announcements. 15:45 - DNS service received incorrect routing information, and users started to experience DNS resolution problems. 4:17 PM - IPv6 connectivity and DNS issue localized and recovery measures taken. Incident re-tagged as Unavailable. 16:18 - according to internal monitoring in Zone A, IPv6 connectivity has been restored. 16:32 - DNS service and load fully functional, incident tagged as Minor again 16: 32–19: 20 - a small part of users experiencing short-term, about a few seconds, losses of network connectivity against the background of elimination of the incident consequences Corrective actionsCorrective actions Already done: • Logic of overlay network agents for Availability Zone A had been changed, ensuring the ability of the overlay network in conditions of overload or short-term controller failure. Thus, the updated logic now works in all zones. This measure prevents the negative effects described both in this incident and in minor incidents related to the loss of network connectivity on May 20 and May 25, 2021. • Routing information exchange logic within the zone is now supplemented with a redundancy mechanism that allows keeping connectivity when the above software error occurs in the BGP controller. This measure also prevents the negative effects similar to those of the June 2 Minor Incident. • Routing information discrepancy detection mechanisms had been improved for faster problem localization and response. In progress: • BGP controller upgrade is in progress in order to eliminate route announcement disappearance error as well as to increase fault tolerance of the routing information exchange scheme in general.