Yandex Cloud experienced a minor incident on April 8, 2022 affecting Object Storage and ru-central1-a and 1 more component, lasting 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 08, 2022, 08:56 AM UTC
There were 5xx errors occurred when writing data to Object Storage.
- resolved Apr 08, 2022, 09:07 AM UTC
Issues were solved at 11:15:16 (GMT +3). All Cloud services are working correctly. We're investigating the causes of the problem.
- postmortem Apr 12, 2022, 07:52 AM UTC
On April 8, during the period from 10:50 to 11:15 Moscow time (GMT +3), some parts of Objects Storage service were unavailable for writing. We apologize to all users affected by the incident and want to share more details about what happened and prevent similar situations in the future. What happened?What happened? On April 7th in 18:50 (GMT +3) Object Storage team has been performing a maintenance to update a scheme inside a service's database. During the update, it was discovered that 0.001% of requests per 5 seconds to database can be processed with errors – this lead to a decision to delay and adjust the process of scheme upgrade. On April 8th in 10:50 (GMT +3) the update procedure was run taking earlier adjustments in consideration. Shortly after, errors were discovered in write logs, and rollback procedure was initiated, which was successfully completed in 11:15. Measures to prevent the recurrence of a similar situation in the futureMeasures to prevent the recurrence of a similar situation in the future • We have discovered the problem in the release process, and revised release procedures.