Yandex Cloud experienced a major incident on September 8, 2020 affecting Certificate Manager and Cloud Functions and 1 more component, lasting 1h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 08, 2020, 11:34 AM UTC
We noticed some problems with the availability of the Identity and Access Management service, which affects other services as well. The developer team is notified and currently investigating this issue.
- identified Sep 08, 2020, 12:17 PM UTC
The Object Storage service is operating correctly, starting from 14:52. The Managed Database services are operating from 15:01. The availability of data in the databases was not violated, only its control access. The Certificate Manager service is operating from 15:15. The Cloud Functions service is available from 15:26. the Yandex Speechkit service is available from 15:31. The service Yandex Message Queue is operating correctly from 15:33.
- resolved Sep 08, 2020, 12:52 PM UTC
The Object Storage service is operating correctly, starting from 14:52. The Managed Databases services are operating from 15:01. The availability of data in the databases was not violated, only its control access. The Certificate Manager service is operating from 15:15. The Cloud Functions service is available from 15:26. the Yandex Speechkit service is available from 15:31. The service Yandex Message Queue is operating correctly from 15:33.
- postmortem Sep 23, 2020, 10:43 AM UTC
On September 8, from 14:03, maintenance work was carried out on the network, leading to some internal short-term unavailability of Yandex.Cloud services. Within minutes, the number of active connections to the Access Service authorization exceeded its capabilities, and it rejected most requests. What happened? To authorize requests to Yandex.Cloud services, a particular internal service is used – Access Service. Yandex.Cloud services use long-lived connections before this service to minimize delays in authorization requests. If the service cannot receive a response from the authorization service for a long time (more than a second), it uses another connection from the pool and re-establishes the previous connection. An error was made while developing a client for the authorization service of one of the Yandex.Cloud services: old connections were not closed during reinstallation and periodically informed the authorization server about their status (a keep-alive mechanism). On September 8, from 14:03, maintenance work was carried out on the network, leading to some internal Yandex's short-term unavailability.Cloud services. Within minutes, the number of active connections to the Access Service authorization exceeded its capabilities, and most requests to it were rejected. Measures to prevent this situation from recurring in the future: We fixed the error in the client to the authorization service and decided to change the architecture of access to it: • Large services will receive their isolated installations of authorization services. This way, problems caused by misbehavior in one service will not affect others. • Authorization services will be protected from attempts to use too many of their resources by one client. We will increase the grouping of authorization services until we implement the specified points so that the resources for processing requests are commensurate with the load that any Yandex.Cloud service can create.