Yandex Cloud incident
SpeechKit, Vision, Translate services issues
Yandex Cloud experienced a major incident on May 8, 2020 affecting SpeechKit and Translate and 1 more component, lasting 1h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 08, 2020, 09:09 AM UTC
We noticed some problems with the SpeechKit service. The developer team is notified and currently investigating this issue.
- identified May 08, 2020, 09:24 AM UTC
During the investigation, the inaccessibility of the Vision and Translate services was confirmed
- resolved May 08, 2020, 09:34 AM UTC
On May 8, from 11:15 to 12:22 (Moscow time), some Yandex.Cloud users encountered errors with AI services queries that returned errors. Speechkit StreamingRecongize was mostly affected. We apologize to all the users affected by this incident and want to tell you more about what happened and measures to prevent similar situations in the future. At 11:00, we carried out technical works to increase the computing power for one of the subsystems through which queries to AI services (gateway) pass. After completion of the works, a problem in the code appeared that is reproduced only on systems with more than one core. After finding the cause of the problem, we began an immediate rollback to the previous configuration. To prevent such problems from happening in the future, we added the AI team's tests to the AI-gateway calculation process. Also, we have increased the list of metrics to follow when rolling out updates.