Platform-wide Service Bus backlog triggers KEDA/AKS control-plane overload

Update timeline

1. investigating Aug 13, 2026, 02:22 PM UTC

   Status: Investigating Nearly every job queue on the platform (20+) has a large simultaneous backlog and DLQ buildup (e.g. loadstratusmodel 740 DLQ, modelversionrelease 573 DLQ, bigdatapublish 684 active/684 DLQ, jsontodocumentdb 340 active/242 DLQ, bim360publish 340 DLQ, loadstratusparts 370 DLQ). KEDA is reacting to all queues at once (ScaledJobs configured up to 100 concurrent replicas), driving 376 Job objects in stratusjobs and 1,277 pods cluster-wide. This volume of simultaneous object creation is overwhelming the AKS control plane (API server/etcd) — kubectl get pods took 6.4s. Control-plane strain is causing scheduling, kubelet pod-sync, and CNI IP allocation (CNS) to hang, so new pods (job pods and DaemonSets: retina-agent, CSI drivers, ama-metrics-node) sit in ContainerCreating/PodInitializing with no IP. Cluster autoscaler is compounding this by adding nodes (6+ new nodes across jobsazlnx03, lnxpdf2, primarylnx03, Windows pool in 15 min), each adding more control-plane load. Result: job pods miss the 1-min peek-lock, KEDA retries, messages exhaust delivery attempts into DLQ — happening simultaneously across queues, not just sendcutlist. Also observed: FailedCreatePodSandBox context deadline exceeded, worker exit from cluster DNS (10.210.0.10) timeout resolving MongoDB, and insufficient-CPU scheduling failures. Affected components Modules (Partial outage)

2. resolved Aug 15, 2026, 07:03 AM UTC

   Status: Resolved This incident has been resolved.