Yandex Cloud incident
Problems with create, delete and edit opertions
Yandex Cloud experienced a minor incident on May 31, 2020 affecting Managed Service for ClickHouse® and Managed Service for MySQL® and 1 more component, lasting 3h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 31, 2020, 03:12 PM UTC
We noticed some problems with operations in the MDB service. Now you cannot change, delete a cluster or create a new one. The developer team is notified and currently investigating this issue.
- identified May 31, 2020, 03:36 PM UTC
We found the cause of the problem and prepared a bugfix. Now we are rolling it out through all the environments.
- identified May 31, 2020, 03:54 PM UTC
We've released the bugfix. The issue was solved at 18:40 (GMT +3). We're restarting failed operations.
- resolved May 31, 2020, 04:10 PM UTC
Failed operations have been restarted and completed.
- postmortem Jun 05, 2020, 04:47 PM UTC
On May 31, we noticed problems with operations in MDB services. Users could not adjust, delete, and create new clusters. We apologize to all users who were affected by the incident. We want to tell you more about what happened and the preventive measures to avoid such situations. What happened?What happened? In mid-May, we released a new version of the Deploy API component for making changes on MDB hosts (earlier, we wrote in more detail about managed database services design on Habr (this post is available in Russian only). Deploy API hosts have Nginx proxy server installed and Go application that processes queries. The release contained a time conversion error, reproduced only after 12:30 UTC on the last day of the month. It caused a panic while querying a specific Deploy API call. On May 31, starting from 15:30 (UTC+3), all queries to create or change clusters were not performed. The monitoring didn't show the error, because Deploy API code response monitoring was configured for the application and not for the Nginx. The application didn't respond 500 and fell in panic. At 17:59, we figured out the issue, and by 18:39, we released a patch that replaced the processing of these situations from panic to errors. Measures to prevent this situation in the future:Measures to prevent this situation in the future: • We fixed the code and rolled out the release with resolved issues. • At the moment, we're working on adding monitoring for Nginx response codes. • We planned to audit the entire code for the use of panic() with error handling where it is not needed.