Yandex Cloud incident

Partial network unavailability

Minor Resolved View vendor source →

Yandex Cloud experienced a minor incident on October 21, 2024 affecting Compute Cloud and DataSphere and 1 more component, lasting 7h 20m. The incident has been resolved; the full update timeline is below.

Started
Oct 21, 2024, 07:30 AM UTC
Resolved
Oct 21, 2024, 02:50 PM UTC
Duration
7h 20m
Detected by Pingoru
Oct 21, 2024, 07:30 AM UTC

Affected components

Compute CloudDataSphereVirtual Private Cloudru-central1-a

Update timeline

  1. investigating Oct 21, 2024, 08:10 AM UTC

    We have noticed occasional short-term network connectivity problems with a number of resources in ru-central1-a and ru-central1-b. It is also possible that clusters of managed services are affected. The developers are already aware of the situation and are working on fixing it.

  2. identified Oct 21, 2024, 08:22 AM UTC

    We are observing the impact on Instance Groups, requests are not processed. Active operations on virtual machine groups have been temporarily suspended.

  3. identified Oct 21, 2024, 09:02 AM UTC

    In the process of solving the problem, we are engaged in load redistribution. As a result, it may be difficult to create new virtual machines in the ru-central1-a zone.

  4. identified Oct 21, 2024, 11:37 AM UTC

    Most of the network problems have been eliminated, we believe that isolated problems may remain; IG works in a limited form (without autoscale and autohealing) in all zones.

  5. identified Oct 21, 2024, 12:55 PM UTC

    The network has been stabilized, and the team continues to eliminate residual phenomena. IG functionality has been fully restored.

  6. identified Oct 21, 2024, 02:42 PM UTC

    The developers fixed the root cause of the incident fixed at 14:35 (Moscow time), the full restoration of connectivity was recorded for all monitoring by 15:45. Now we continue to work to stabilize the situation and prevent relapses.

  7. resolved Oct 21, 2024, 02:50 PM UTC

    During the period 15:45-17:50 (Moscow time), we do not observe any regressions, the technical team has completed corrective measures.

  8. identified Nov 02, 2024, 02:30 PM UTC

    Summary of the series of network connectivity incidents in zone ru-central1-a is provided in the incident description (https://status.yandex.cloud/en/incidents/972) dated October 16, 2024.