Yandex Cloud incident

Labels issues in Managed Service for Kubernetes®

Yandex Cloud experienced a minor incident on March 3, 2021 affecting Managed Service for Kubernetes® and ru-central1-a and 1 more component, lasting 1d 2h. The incident has been resolved; the full update timeline is below.

Started: Mar 03, 2021, 06:37 PM UTC
Resolved: Mar 04, 2021, 09:00 PM UTC
Duration: 1d 2h
Detected by Pingoru: Mar 03, 2021, 06:37 PM UTC

Affected components

Managed Service for Kubernetes®ru-central1-aru-central1-b

Update timeline

investigating Mar 04, 2021, 08:16 AM UTC

We noticed problems with system labels and labels set via the service API. The team is already fixing it. Until then, it is not recommended to use such labels to control pods.
identified Mar 04, 2021, 06:01 PM UTC

An update has been released for all clusters to prevent new incorrect node labels from appearing. The second part of the fix is planned soon. It will correct the wrong node labels that appeared during the incident.
resolved Mar 05, 2021, 03:17 PM UTC

As a result of the incident, labels assigned through the Managed Kubernetes API to individual groups of nodes could be assigned to nodes in other groups in the cluster. We decided not to automatically restore incorrectly placed labels because this can lead to an unwanted transfer of loads between the cluster nodes. You can remove erroneous labels from cluster nodes through the Kubernetes API. For this, you can use the kubectl utility. We will send the affected clusters' owners detailed instructions on the actions necessary to restore the labels' consistent state painlessly. We will mark problem clusters with a unique cluster label yc_k8s_incorrect_labels, and we will display a warning for such clusters in the Yandex.Cloud web console.