Yandex Cloud experienced a minor incident on April 7, 2021 affecting Compute Cloud and Managed Service for Apache Kafka® and 1 more component, lasting 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 07, 2021, 12:39 PM UTC
We noticed problems with the availability of some services among some users. The developers are already in the know and are fixing it.
- resolved Apr 07, 2021, 12:53 PM UTC
The issue was solved at 15:50 (GMT +3). All cloud services are working correctly.
- postmortem Apr 21, 2021, 11:54 AM UTC
On April 7 at 14:33 Moscow time, there were problems with the of the data center module, which led to a reboot of the server and network equipment in several stacks. The problem was resolved at 15:50. We apologize to all users affected by this incident, and would like to give more detail about it and the measures we have taken to prevent a recurrence of such situations in the future. What happened?What happened? On April 7 at 14:33 Moscow time, a reboot of the server and network equipment in several stacks because of problems with the power supply led the disks’ IO to freeze. The operations read/write/pread/pwrite/etc. did not respond, and the processes querying remained in a D state until the correction of the problem. On the MDB server side, clusters using the NBS disk under the master or a synchronized replica were affected. Disk operations throttled, however the database could work with read operations. At 15:50, the problem had been resolved, and all Yandex.Cloud services were functioning as usual. Measures to prevent a recurrence of the situation in the future:Measures to prevent a recurrence of the situation in the future: A special additional inspection of the data center’s electrical equipment was carried out, which resulted in the creation of a list of measures for the improvement of module servicing.