Yandex Cloud experienced a major incident on August 21, 2024 affecting DataLens and ru-central1-a and 1 more component, lasting 4h 40m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 21, 2024, 05:27 AM UTC
We have noticed issues with loading datasets for some users. The service team is already aware of the situation.
- resolved Aug 21, 2024, 08:39 AM UTC
At 11:30 MSK, the problem was resolved. Datasets functionality has been fully restored.
- postmortem Sep 27, 2024, 02:54 PM UTC
On August 21, from 06:55 to 11:25 MSK, the DataLens service was partially unavailable. Some user requests for loading connections, datasets, and data from sources were executed with errors. We apologize to all users affected by this incident. Here are the details of what happened and the measures being taken to prevent such situations in the future. Cause of the Incident: When working with sources, DataLens uses a separate service deployed in a Kubernetes cluster. At 06:50 MSK, during the application of updates to the Kubernetes cluster, the Linux kernel version on the nodes was updated. An error in packet processing when redirecting them through the vxlan technology to another cluster node was introduced in the new kernel version. After applying the update, some user requests that required redirection between nodes started to fail. Preventative Measures for the Future: • Enhance alerting mechanisms by the DataLens team to monitor issues with request execution between internal services • Change the Kubernetes cluster's automatic update procedures • Improve network interaction testing mechanisms by the Managed Kubernetes service • Explore additional redundancy for k8s-related resources