Yandex Cloud experienced a minor incident on June 22, 2023 affecting Managed Service for Kubernetes® and ru-central1-a and 1 more component, lasting 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 28, 2023, 05:12 PM UTC
22/06 at 14:45 UTC+3, we noticed a problem with the unavailability of the k8s master nodes.
- resolved Jun 28, 2023, 05:13 PM UTC
22/06 at 15:15 UTC+3 this problem was fixed. We will provide details later.
- postmortem Jul 10, 2023, 10:31 AM UTC
What happened?What happened? Partial unavailability of k8s masters. Timeline:Timeline: • June 22 14:40 (UTC+3) - an overload of balancers followed by the unavailability of KMS; • June 22 14:45 (UTC+3) - a problem with the unavailability of k8s masters was detected; • June 22 15:15 (UTC+3) - the problem was fixed. Reasons and detailed description:Reasons and detailed description: The reason for the unavailability of KMS was the protective shutdown of balancing due to overload. The unavailability of k8s was caused by disabling of request balancing to k8s masters due to the lack of positive health-checks, which was caused by the unavailability of KMS. Measures to prevent such incidents in the future:Measures to prevent such incidents in the future: • To improve the mechanism for collecting diagnostic information from KMS balancers in order to detect future congestion problems in advance; • To adjust the health-check mechanism of k8s masters to reduce the effect of KMS on their outcome.