Problems with Managed Service for Kubernetes®

Update timeline

1. investigating Mar 03, 2021, 02:27 PM UTC

   We noticed some problems with Managed Service for Kubernetes®. Developer team is notified and currently investigating this issue.

2. identified Mar 03, 2021, 03:41 PM UTC

   Unexpected problems are related to the group nodes updating. We continue to work with the problem.

3. identified Mar 03, 2021, 04:05 PM UTC

   Right now, you can obtain the most up-to-date information about the state of nodes and pods through the kubectl tool; for now, focus only on its output. There is a delay in the console and CLI.

4. identified Mar 03, 2021, 05:13 PM UTC

   The majority of clusters are now functioning properly. If you still face problems with labels, please contact us: https://console.cloud.yandex.com/support/

5. resolved Mar 03, 2021, 06:37 PM UTC

   All Managed Kubernetes clusters are now available. We still have issues with managing labels that are set using API. Hotfix is on it's way.

6. postmortem Mar 12, 2021, 03:20 PM UTC

   On March 3 at 4:10 pm Moscow time, after the planned release, we faced with an unforeseen process of updating the nodes on the groups of nodes of the Yandex Managed Service for Kubernetes® service with set service labels or cloud labels. The labels specified for one node group have been assigned to all nodes in the cluster. Incorrect placement of labels in some clusters led to the fact that there were not enough resources on the user's nodes for certain types of load. As a result, the autoscaling of host groups was disrupted, and applications running in such clusters began to experience availability issues. Besides, the state of the nodes was displayed incorrectly in the console and API. We apologize to all users affected by this incident. We want to tell you more about what happened and what corrective measures we're planning to implement in order to prevent a similar situation from recurring in the future. What happenedWhat happened Yandex Managed Service for Kubernetes® has added support for changing labels in already created groups of nodes using the API; before that, you could only set labels when creating a group. This update changed the logic for generating the configuration of nodes specified through virtual machine groups' metadata. Also, the lists of labels of nodes managed by the backend of the service have been expanded. We made an error in the new logic of the formation of metadata, due to which the service metadata of all groups of nodes with set labels changed. It led to an update of the groups of virtual machines' specifications and launched their redeployment process. A unique set of tests verifies that any change in the code does not lead to a configuration change. It ensures that virtual machines configurations in a group of nodes already available to the user remain unchanged. But the existing set of tests did not reveal any problems with the new logic. An unscheduled upgrade of site groups can result in only unreplicated pods being unavailable: the system, by default, ensures that sites are gradually upgraded. During the upgrade process, the number of nodes required by the specification is always available in the group. The old nodes are settled, taking into account the Pod Disruption Budget. However, during the release process, we made another mistake: the controller of each group of nodes in the cluster, which controls virtual machines, tried to set its labels on all cluster nodes. There were not enough resources for the nodes that the user allocated for certain types of the load because after the update, there were other pods there. Moreover, if the names of the labels of the groups of nodes in the cluster coincided, then the controllers of different groups of nodes began to compete for the label's value. The label was constantly changing its value, which destabilized the work of the cluster load scheduler. The error affected the system label, by which the cluster dashboard in the console, the node status API and the Cluster Autoscaler match the node and the node group that belongs to it. It led to the fact that many nodes in the API and UI were displayed in the "not connected" status, although they were connected to the cluster and worked correctly. Cluster Autoscaler could not match the created nodes with groups; therefore, a short time after creating a new node, it deleted it, which led to a problem with cluster autoscaling. Any significant change in the node group controllers' workflows during release goes through a "long" release cycle when the clusters are gradually transferred under the new backend's control under the service team's control. In particular, we check that the transfer of control does not lead to the redeployment of groups of nodes. If problems are found, we quickly the cluster to the previous version. In this release, the core process logic has not changed, and the change in the label management logic has been minimal. Therefore, considering that the tests of the immutability of configurations did not reveal any problems, we decided to release according to the scheme when the new backend immediately gains control over all clusters. ChronologyChronology March 3 • 16:10 – the release of the Yandex Managed Service's new functionality for Kubernetes® service, containing an incorrect code, has been completed. • 16:17 - backend metrics show an increase in the load on the storage cluster of the service. It is unusual but expected – after a restart, the backend fills the client cache. • 16:40 - metrics have problems with requests to the service API. The service team decides to add resources to the storage cluster. • 17:00 - storage cluster expansion is completed, API responses return to normal. • 17:20 - An unscheduled update of host groups has occurred. • 17:40 - we discuss the possibility of rolling back the release and test the hypothesis that the rollback will lead to the reverse redeployment of the already updated groups. • 18:09 - a problem with the placement of labels. Many customers complain that most of the nodes in the clusters are unavailable, but the Cluster API's state suggests otherwise. • 18:30 - a check confirms that a rollback will lead to a reverse deployment of all updated groups, and the clusters that have completed the update are working as usual. The team decides not to roll back the release but to sort it out point by point, guided by the support work regulations. It turns out that the unavailability problems are related to the lack of resources on the nodes with the correct labels. The support team helps customers reconfigure their clusters to make sure there is enough space for the service. • 19:00 – the status informs our users in the dashboard in the cloud console, which displays information incorrectly. We place a recommendation to use kubectl to obtain up-to-date information about the state of nodes on the page with the console components' status. Several clients who have had problems report that all critical pods have started and that the applications have been restored to work. • 19:30 - the support team continues to work on calls and help restore applications by reconfiguring clusters and pods. The service team continues to investigate the cause of the incorrect label behaviour. • 20:13 — application work in most clusters is restored. • 20:59 - we investigate the reason for the incorrect operation of the labels. • 21:37 - the support team sends recommendations on restoring applications in clusters in case of problems with the labels. • 23:31 - we investigate the reason for the unexpected update of the host groups. March 4 • 01:04 - we discuss options for fixing the problems. • 03:48 - we confirm all hypotheses about the causes of the incident and prepare a plan for correcting errors. • 10:16 - we finalize the plan. • 11:36 - 13:50 - the release of the update with bug fixes and its testing is in the pre-production. • 13:54 - 20:50 - the release of the update with bug fixes is in production. First of all, we restore the labels to work for the clients who reported the problem. The labels on node groups of all clusters work correctly, but some nodes still have incorrect labels. Autoscale health and Node State API restored for all clusters. March 4-5 We prepare a plan for correcting the remaining incorrect labels. The decision is made not to restore custom labels to the correct state automatically. It can lead to unwanted workload transfers between nodes and possible application availability problems if resource problems are in the clusters. A special label is added to the affected clusters, indicating that there are problems with the nodes' labels, and special instructions are prepared to fix them (either independently or with the help of the support team). Measures to prevent this situation from recurring in the futureMeasures to prevent this situation from recurring in the future • We expanded the scope of the "long" release cycle. We developed the checks performed to cover all the functionality of the Node Group Controller. Any changes to the configuration of groups of nodes will be applied to the cluster park one at a time, starting with the rapid release channel. Validation checks will check the cluster nodes' whole state before and after the release. If problems appear on any of the clusters, the release will stop. • For virtual machine groups managed by host groups, changes to the metadata containing host configuration no longer result in virtual machine redeployment. Even if a mistake is made and the host groups' configurations change unexpectedly, the virtual machine group will not redeploy but update the metadata, which will not affect the already created hosts. • We will improve tests for the immutability of configurations. The tests did not fully consider how the configuration should behave when changing user-generated data (such as tags). We will add such cases and auto-generated scripts covering a more extensive set of combinations of input parameters. • The configurations of test clusters in the pre-production environment will be supplemented so that with future updates, we can identify such problems immediately. Revisiting and developing these configurations will be a mandatory step when developing new functionality. CompensationCompensation Customers affected by the incident can contact support within a month, specifying the affected clusters' id. We will make estimates and calculate compensation in the form of grants for Yandex Managed Service for Kubernetes® and Yandex Compute Cloud.