Stratus incident
Modules dashboard intermittent outage — node-churn causing Valkey/mesh connectivity errors
Resolved after 4 hours 27 minutes, on August 11, 2026.
Impact: modules-dashboard.prd.gtpstratus.com intermittently returning 503 ("Micro-Frontend Unavailable" in the Stratus host) since ~18:35 UTC. Currently sustained, not self-resolving after 15+ min of monitoring. Root cause chain (working theory, not yet fully confirmed): 1. A burst of unrelated Stratus core job workload (stratusjobs/stratus/stratusapi namespaces — importbundle, sendcutlist, runpdfreport, etc.) triggered the cluster autoscaler to rapidly add nodes to the shared AKS node pools (aks-jobsazlnx03, aks-primarylnx03) that Modules also runs on. 2. modules-dashboard and one Valkey pod (valkey-0) got rescheduled onto newly-joined nodes during the churn. 3. Istio/CNI routing convergence for pods on brand-new nodes appears to be taking longer than expected, causing "no route to host" both on modules-dashboard's ingress path and on modules-backend's connections to Valkey (port 6379, PassthroughCluster — 15-20 failed connection attempts per 25s window, sustained). 4. Side effect: metrics-server was also disrupted mid-restart during the churn, causing unrelated HPA "FailedGetResourceMetric" noise across the cluster (istiod, istio-ingressgateway, App Config) — cosmetic, not part of the outage. Ruled out: this does NOT look like the Valkey Sentinel stale-ghost-entry bug fixed yesterday (PLAT-1879/PR #548) — that was permanently-dead entries never pruned; this is fresh/transient unreachability tied to active pod rescheduling. Separate, likely unrelated minor issue: modules-backend logged real 400s on assembly-tracking-status and package-tracking-status around 18:35, reported by a user updating a package's tracking status in the Viewer. Worth investigating on its own but doesn't explain the dashboard/Valkey connectivity failures. Status as of declare: still degraded — dashboard failing ~2/3 checks, Valkey still throwing ~20 connection errors per 25s window. Not converging on its own.
Update timeline
- investigating Aug 11, 2026, 06:55 PM UTC
Impact: modules-dashboard.prd.gtpstratus.com intermittently returning 503 ("Micro-Frontend Unavailable" in the Stratus host) since ~18:35 UTC. Currently sustained, not self-resolving after 15+ min of monitoring. Root cause chain (working theory, not yet fully confirmed): 1. A burst of unrelated Stratus core job workload (stratusjobs/stratus/stratusapi namespaces — importbundle, sendcutlist, runpdfreport, etc.) triggered the cluster autoscaler to rapidly add nodes to the shared AKS node pools (aks-jobsazlnx03, aks-primarylnx03) that Modules also runs on. 2. modules-dashboard and one Valkey pod (valkey-0) got rescheduled onto newly-joined nodes during the churn. 3. Istio/CNI routing convergence for pods on brand-new nodes appears to be taking longer than expected, causing "no route to host" both on modules-dashboard's ingress path and on modules-backend's connections to Valkey (port 6379, PassthroughCluster — 15-20 failed connection attempts per 25s window, sustained). 4. Side effect: metrics-server was also disrupted mid-restart during the churn, causing unrelated HPA "FailedGetResourceMetric" noise across the cluster (istiod, istio-ingressgateway, App Config) — cosmetic, not part of the outage. Ruled out: this does NOT look like the Valkey Sentinel stale-ghost-entry bug fixed yesterday (PLAT-1879/PR #548) — that was permanently-dead entries never pruned; this is fresh/transient unreachability tied to active pod rescheduling. Separate, likely unrelated minor issue: modules-backend logged real 400s on assembly-tracking-status and package-tracking-status around 18:35, reported by a user updating a package's tracking status in the Viewer. Worth investigating on its own but doesn't explain the dashboard/Valkey connectivity failures. Status as of declare: still degraded — dashboard failing ~2/3 checks, Valkey still throwing ~20 connection errors per 25s window. Not converging on its own.
- resolved Aug 11, 2026, 11:22 PM UTC
This incident has been resolved.
Your product runs on more than Stratus
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services