Yandex Cloud experienced a major incident on March 30, 2025 affecting Application Load Balancer and Cloud Interconnect and 1 more component, lasting 11h 35m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 30, 2025, 09:47 AM UTC
Our team is investigating the circumstances.
- identified Mar 30, 2025, 10:06 AM UTC
At the moment, the ru-central1-b zone is completely inaccessible due to power issues. The team is currently investigating the causes and working to resolve them. We recommend that clients shift their workload to other zones. We will provide updates around 13:30.
- identified Mar 30, 2025, 10:36 AM UTC
• The ru-central1-b zone is completely unavailable due to a power supply failure. The team is investigating the causes and working to resolve the issue. • At the moment, the cloud team has completely disabled NLB & ALB load balancing in the ru-central1-b zone to ensure that the zone's recovery is under our control. • We recommend that clients move the load to other availability zones. If you experience problems switching/moving managed services, please create a ticket in technical support. The Yandex Cloud team will assist promptly. • Status update at 14:30.
- identified Mar 30, 2025, 11:42 AM UTC
• The ru-central1-b zone is completely without power. There is a failure with the power supply system, and there is no power from the supplying substation. The team is in contact with the power supply organization and is currently awaiting the restoration of power. • At the moment, the cloud team has completely disabled NLB & ALB balancing in the ru-central1-b zone to ensure that the zone's recovery is under our control. • We recommend that clients move the load to other availability zones. The cloud support team is assisting with switching/moving managed services within the tickets. • ETA for the zone recovery is 1-3 hours after full power restoration. • Status update at 15:30.
- identified Mar 30, 2025, 12:56 PM UTC
• Since 15:30, we have been observing the start of the power supply restoration to the data center. ETA for full power restoration is 1 hour. The plan for restoring the ru-central-1b zone: • Restore the basic infrastructure. • Restore the Compute service. • Return zone B to the NLB/ALB load balancers. • ETA for restoring Compute services is 1-2 hours after the full restoration of power to the data center. • Status update at 16:45, we also plan to provide separate updates according to the zone recovery plan.
- identified Mar 30, 2025, 02:04 PM UTC
• Power supply to the data center has been restored, all equipment is running. • Starting the restoration of the Compute service. ETA is 1–2 hours. • The load will be returned to the zone via NLB and ALB services after the Compute restoration, we will inform with a separate update. • Status update at 18:00.
- identified Mar 30, 2025, 03:06 PM UTC
• Power supply to the data center has been restored, all equipment is running. • The basic infrastructure has been restored. • The cloud team continues to work on restoring the Compute service. ETA for restoring user VMs is 1 hour. • Along with the restoration of user VMs, we expect the start of the restoration of Managed DBs. • We plan to return the load to the ru-central1-b zone via NLB and ALB services after the complete restoration of Compute; we will inform with a separate update. • The next status update will be at 19:00.
- identified Mar 30, 2025, 04:08 PM UTC
We have started deploying the first VMs (Compute) in the ru-central1-b zone, and we will be updating the status every hour. We are working to complete these operations as quickly as possible.
- identified Mar 30, 2025, 05:11 PM UTC
Currently, we have restored approximately 15% of the load in zone B. Based on the current recovery rate, we expect full recovery by 21:30. We are continuing to work on accelerating the process.
- identified Mar 30, 2025, 06:19 PM UTC
• 82% of virtual machines in ru-central1-b are up and running (including all virtual machines not under Instance Groups). • The team is working on restoring the remaining VMs managed by Instance Groups as quickly as possible. ETA — current estimate: 1 hour. • Managed Kubernetes: issues with the autoscaler should be resolved after restoring VMs managed by Instance Groups. • Compute: there is currently increased load on Compute operations; the "Operation queue limit exceeded" error is expected, and if it occurs, you should wait; there is no need to change the quota for the number of operations. • ALB & NLB: we still do not open zone B, we will return with the next update. Next update at 22:00.
- identified Mar 30, 2025, 07:22 PM UTC
• All virtual machines in ru-central1-b are up and running (including VMs under Instance Groups). • Compute & Instance Groups: there may be delays in operations, including operations with Managed Kubernetes nodes. • NLB: we are starting to return traffic to zone B. • ALB: we are not returning traffic to zone B yet. • If you experience problems with individual services, please contact technical support. Next update at 23:15.
- identified Mar 30, 2025, 08:25 PM UTC
• Compute & Instance Groups & K8S: services are operating normally, occasional delays in Compute operations may be observed. • MDB: the service is operating normally, automation is continuing to restore affected replicas. • NLB: the service is operating normally, traffic has been returned to zone B. • ALB: we are returning traffic to zone B and monitoring its performance. • If you experience problems with individual services, please contact technical support. Next update at 00:15.
- resolved Mar 30, 2025, 09:10 PM UTC
As of 00:00, we believe that the incident has been largely mitigated: • Compute & Instance Groups & K8S: services are operating normally. • MDB: the service is operating normally, automation is continuing to restore affected replicas. • NLB & ALB: services are operating normally, traffic has been returned to zone B. If you experience problems with individual services, please contact technical support. We apologize sincerely for this incident. We will provide information about the causes of the incident and subsequent actions later in the week.