Stratus incident
Platform-wide Service Bus backlog triggers KEDA/AKS control-plane overload
Resolved after 1 day 16 hours, affecting Modules, on August 13, 2026.
Nearly every job queue on the platform (20+) has a large simultaneous backlog and DLQ buildup (e.g. loadstratusmodel 740 DLQ, modelversionrelease 573 DLQ, bigdatapublish 684 active/684 DLQ, jsontodocumentdb 340 active/242 DLQ, bim360publish 340 DLQ, loadstratusparts 370 DLQ). KEDA is reacting to all queues at once (ScaledJobs configured up to 100 concurrent replicas), driving 376 Job objects in stratusjobs and 1,277 pods cluster-wide. This volume of simultaneous object creation is overwhelming the AKS control plane (API server/etcd) — kubectl get pods took 6.4s. Control-plane strain is causing scheduling, kubelet pod-sync, and CNI IP allocation (CNS) to hang, so new pods (job pods and DaemonSets: retina-agent, CSI drivers, ama-metrics-node) sit in ContainerCreating/PodInitializing with no IP. Cluster autoscaler is compounding this by adding nodes (6+ new nodes across jobsazlnx03, lnxpdf2, primarylnx03, Windows pool in 15 min), each adding more control-plane load. Result: job pods miss the 1-min peek-lock, KEDA retries, messages exhaust delivery attempts into DLQ — happening simultaneously across queues, not just sendcutlist. Also observed: FailedCreatePodSandBox context deadline exceeded, worker exit from cluster DNS (10.210.0.10) timeout resolving MongoDB, and insufficient-CPU scheduling failures.
Affected components
Update timeline
- investigating Aug 13, 2026, 02:22 PM UTC
Nearly every job queue on the platform (20+) has a large simultaneous backlog and DLQ buildup (e.g. loadstratusmodel 740 DLQ, modelversionrelease 573 DLQ, bigdatapublish 684 active/684 DLQ, jsontodocumentdb 340 active/242 DLQ, bim360publish 340 DLQ, loadstratusparts 370 DLQ). KEDA is reacting to all queues at once (ScaledJobs configured up to 100 concurrent replicas), driving 376 Job objects in stratusjobs and 1,277 pods cluster-wide. This volume of simultaneous object creation is overwhelming the AKS control plane (API server/etcd) — kubectl get pods took 6.4s. Control-plane strain is causing scheduling, kubelet pod-sync, and CNI IP allocation (CNS) to hang, so new pods (job pods and DaemonSets: retina-agent, CSI drivers, ama-metrics-node) sit in ContainerCreating/PodInitializing with no IP. Cluster autoscaler is compounding this by adding nodes (6+ new nodes across jobsazlnx03, lnxpdf2, primarylnx03, Windows pool in 15 min), each adding more control-plane load. Result: job pods miss the 1-min peek-lock, KEDA retries, messages exhaust delivery attempts into DLQ — happening simultaneously across queues, not just sendcutlist. Also observed: FailedCreatePodSandBox context deadline exceeded, worker exit from cluster DNS (10.210.0.10) timeout resolving MongoDB, and insufficient-CPU scheduling failures.
- resolved Aug 15, 2026, 07:03 AM UTC
This incident has been resolved.
Your product runs on more than Stratus
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services