Yandex Cloud experienced a minor incident on January 31, 2023 affecting Data Processing and Data Transfer and 1 more component, lasting 7h 40m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 31, 2023, 04:00 AM UTC
We noticed problems with the availability of Yandex Cloud services' API among some users. The developers are already aware about this problem and now they are fixing it.
- resolved Jan 31, 2023, 08:19 AM UTC
All Cloud services are working correctly. We will provide details later.
- postmortem Feb 08, 2023, 02:00 PM UTC
On January 31, from 03:15 to 10:45 Moscow time, some Yandex Cloud users noticed problems with MDB API: periodically, managing clusters and backups API calls responded with errors, and operations themselves took longer than usual. We apologize to the users affected by the incident. We want to tell you more about what happened and what measures well we take to prevent similar situations in future. What happened?What happened? MDB API sends requests managing clusters and backups to the same service as background operations on the cluster. At 03:15, the MDB API increased the load on background operations, which led to increasing service response time. An outdated retry algorithm in background operations service caused an avalanche of retries when service slowed down. This particular case wasn't covered by alerting service, so we haven't noticed the problem immediately. In the period from 7:00 to 10:30, we took measures to eliminate the error, at 10:45 — the API began to work normally. Measures to prevent the recurrence of this incidentMeasures to prevent the recurrence of this incident • The coverage of alerts in the part of the internal API has been expanded in order to quickly detect problems of this kind. • Added limits on the number of requests in order to cope with congestion. • A modern policy of retries in load sources has been implemented in order to not increase the load on the service when it stops coping with it. • The load from user and background operations was divided into different services so that requests from background operations would not affect requests from users.