Yandex Cloud experienced a major incident on June 9, 2020 affecting Cloud Functions and Container Registry and 1 more component, lasting 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 09, 2020, 12:25 PM UTC
We noticed some problems with the Object Storage service. There is also an impact on the performance of Cloud Functions. The developer team is notified and currently investigating this issue.
- resolved Jun 09, 2020, 01:11 PM UTC
We found and fixed the cause of the problem. Further details will be provided upon completion of the investigation.
- postmortem Jun 26, 2020, 12:44 PM UTC
On June 9, from 15:07 to 15:54 (Moscow time), Object Storage was unavailable for writing, and from 15:26 to 15:54 for reading. What happenedWhat happened On June 9, at 12:00, the Object Storage team had technical works to switch the load to a new version of the application, which would provide faster and more reliable work with the service. The first step of switching is to set a flag on client clouds, which manages the task of controlling the new version being served to all new buckets. The second step is to switch the already created buckets to the new version. For this, each bucket, in turn, is briefly set to the read-only mode, then it's transferred to a new version and exits the read-only mode. The process of switching the bucket takes only a few seconds. For the second step to work consistently, the old version of the application checked whether the bucket is in a read-only state for each modifying request (PUT / POST / DELETE). This check did not cause problems during the previous switchings, but on June 9 at 15:07, it led to an increased load on the table with a list of available buckets. It led to the Object Storage service becoming temporarily unavailable for writing. After a few minutes, it also became unavailable for reading. We solved the problem at 15:54. Measures to prevent the recurrence of a similar situation in the future:Measures to prevent the recurrence of a similar situation in the future: We optimized the operation of queries and storing a table with a list of buckets.