Disk Issues

Update timeline

1. investigating Jun 19, 2020, 08:09 AM UTC

   We noticed a problem when working with disks, there may be problems when reading / writing disks, when starting and restarting virtual machines and hosts of MDB clusters. The developer team is notified and currently investigating this issue.

2. identified Jun 19, 2020, 10:46 AM UTC

   Today, from 8:40 to 10:15 in the ru-central-b availability zone, there was a hardware failure of the equipment on four physical servers. The incident affected less than 1% of the physical disks located in this zone. It's an unlikely event, the causes of which we're investigating at this moment. In Yandex.Cloud, data is stored on replicated network drives. The network drive can withstand simultaneous failures of two servers without data loss. At the time of the failure, data begins to replicate to other servers. But the simultaneous failure of 4 servers in such a short period led to the data not managing to be replicated. As a result, some of the data on disks have been lost. During such failure, the virtual machine "freezes," and manual intervention is required to fix it. We're currently working on rebooting virtual machines without lost data blocks. We are now contacting those customers affected by the incident to help them deal with the consequences of the problem individually. If you see that this incident affected you, please contact Yandex.Cloud support, or your manager.

3. identified Jun 19, 2020, 01:57 PM UTC

   We continue to investigate the problem of the incident. Notifications and recommendations have been sent to all affected users.

4. resolved Jun 19, 2020, 06:09 PM UTC

   Currently, all services are operating as usual, both for creating and changing Cloud resources in the ru-central-b zone. The status of those resources that were affected during the incident: • Managed Service for Kubernetes is fully restored without data loss. • Managed Service for PostgreSQL and MySQL are in the process of recovery without data loss. • 23% of the affected clusters without redundancy (on the same host) of the Managed Service for ClickHouse, Redis, and MongoDB services have been restored from the last backup. The rest of the cluster is in the process of recovery without data loss. For Compute Cloud users affected by the incident: • Use our prepared recommendations on emergency recovery of disks. • In the Cloud console, affected disks are marked with a unique plate so you can instantly identify them on the page. We apologize to everyone affected by this incident. Shortly we will prepare a detailed report where we'll tell in more detail what happened and measures to prevent a similar situation's recurrence in the future.

5. identified Jun 22, 2020, 10:22 AM UTC

   On June, 22 at 1:30 MSK all affected database clusters (Managed Service for PostgreSQL, MySQL, MongoDB, ClickHouse and Redis) were successfully restored from backups.

6. postmortem Jun 26, 2020, 04:19 PM UTC

   On Friday, June 19, some clients wholly or partially couldn't access Compute Cloud, Managed Service for ClickHouse, Managed Service for Kubernetes®, Managed Service for MongoDB, Managed Service for MySQL®, Managed Service for PostgreSQL, Managed Service for Redis™ in the ru-central-b availability zone. We localized the problem at 10:15, after which our team specifically helped users eliminate the consequences of the incident. We apologize to all the users affected by this incident. We want to tell you more about what happened and measures to prevent a similar situation in the future. What Happened?What Happened? In Yandex.Cloud, data is stored in a proprietary distributed network storage system. A network drive can withstand the simultaneous failure of two servers from its segment without data loss—the size of the failure segment used in Yandex.Cloud is a group of servers located in different server racks. Each server uses one physical disk to store data for a particular network drive. When a physical disk or server fails, data begins to be replicated to other servers in this segment until the replication rate of each data block is restored. On June 19, an unlikely situation occurred when four servers failed, three of which were in the same failure segment. It led to the fact that not all data managed to be replicated. As a result, part of the data on the disks was lost, and the services listed above failed. The course of events: 8:40 - Failure of the first server, and we recognized it as unrecoverable. The situation is regular. 9:48 - We recognize the second server of the same segment as unrecoverable. Immediately after that, an automatic system triggers a warning about the threat of data loss. The service team quickly begins to search for the problem, since this is a dangerous situation, and data loss becomes possible if a third server in the segment soon fails. 10:09 - The third server from the same failure segment fails. It's not possible to localize the problem before the failure of the third server. Immediately after this, we forcibly stop the automation on all Yandex.Cloud servers. 10:15 - The server fails in another failure segment, which we mention in the initial message about the problem. But since this is the first non-working server in another segment, nothing happens to the data in this segment. Besides, we already stopped the automation, and the server returns to service. 10:15 - We begin to assist affected users at their request. Implications for users: • In Compute Cloud, the failure affected part of the user drives located in the ru-central-b zone. Partially the data on these disks were lost, which led to a partial or complete loss of the disk. We sent recommendations to affected users to perform a disk recovery according to the instructions. We marked all affected disks in the Yandex.Cloud console. At the same time, we were able to recover part of the disks on our own without users' participation, where the internal system part of the disk turned out to be affected. Later we removed the mark from such disks. If you don't see it in the console, then no action is required. • Managed databases with clusters deployed with network drives in several availability zones, were unavailable for writing for several minutes. There was no data loss. Such inaccessibility is an abnormal situation since the processing of partial disk loss wasn't provided. This system's behavior will be improved, and switching due to long-term problems with the disk will be carried out in tens of seconds. • The managed databases Managed Service for PostgreSQL and Managed Service for MySQL®, whose clusters were deployed only in the affected ru-central-b availability zone, were restored without losing data from the last backup but with the service unavailability for the recovery time. • We restored the managed database resources Managed Service for ClickHouse, Managed Service for MongoDB and Managed Service for Redis™ clusters deployed only in the affected ru-central-b availability zone from the last backup with data loss for the last business day. • The incident affected only a few users of the Managed Service for Kubernetes, who used a non-fault tolerant type of the master. During the event, Kubernetes was unavailable for the control plane, while the client services running in the clusters continued to function normally. We restored these masters from the latest backup. The reasonsThe reasons The reason for this failure was initially mistakenly identified by us as exclusively a hardware failure since it occurred at the level of our basic infrastructure. Automation in our data centers independently finds possible physical equipment problems, after which it decides on how to fix the situation. Depending on the issue, the automated system may give instructions for replacing the disk, rebooting the server, or completely reconfiguring the server if other measures did not help. During this process, Yandex. Cloud users don't notice server or disk failures, and for them, all services continue to work as usual: virtual machines and data are automatically transferred to other servers. Our internal investigation revealed that there was an error in the automatic procedure for fixing hardware problems. In the unlikely situation with issues with a specific disk, the automation concluded that the problem affects the entire server, or rather, all its disks, and not just one disk, as it was. The system rebooted the server, detected the problem again and sent it to a complete reconfiguration. Attempts to reconfigure were unsuccessful, and as a result, an extreme measure was taken – the recovery procedure formatted all disks. Incorrect determination of the cause of the physical infrastructure problem didn’t lead to visible consequences for the user. Sending a "healthy" server to a full reconfiguration with all disks is unnoticeable to users and ordinary. But since in this situation, there was a simultaneous failure of three servers, while the Yandex.Cloud failure model contained a simultaneous failure of two servers in a segment; the data on the formatted disks couldn’t replicate and were eventually lost. Measures to prevent the recurrence of a similar situation in the future:Measures to prevent the recurrence of a similar situation in the future: • We've already fixed the error in the automatic procedure, which incorrectly determined the hardware problem's level. • Before sending the server or disk to a full reconfiguration for any reason, we added a mandatory delay (one day). It will allow engineers to manually handle the situation and return physical disks to the cluster without data loss. • We’ll add an extra verification level – now the data storage system in Yandex.Cloud will confirm any action with the equipment. We can postpone work with a failed physical disk or server for any necessary period until we're sure of the safety of this action. In the incident that occurred, this would allow not to send the second and third servers for reconfiguration, but to freeze them until we restore the first server. • We’ll introduce the mandatory backup of the Managed Service for Kubernetes masters several times a day. Recommendations for our clients: • To increase the fault tolerance of applications, use our cross-zone solutions, or have a backup copy of critical data in other Yandex.Cloud availability zones. • To work out a system of active monitoring of your service, hosted on our platform, to increase control and accessibility of your service. If you need help with this, we're always ready to help you choose the right partner or set up monitoring yourself by contacting the support service. What we recommend our customers do: • Despite all the measures taken, one availability zone (AZ) is a likely failure domain, which, although with a very low probability, can fail. Therefore, we recommend that Yandex.Cloud clients use either our cross-AZ solutions or have a backup copy of critical data in other AZs. • If it's necessary to withstand a service failure in one AZ, we recommend working out an active monitoring system for your service located on the platform to increase control and accessibility of your service, with architects or integration partners of Yandex.Cloud.