Is Stratus down?

Status: Investigating Users cannot sign in to Stratus (www.gtpstratus.com "www.gtpstratus.com": http://www.gtpstratus.com). Sign-in attempts end on a "There was a problem signing you in" error page. The problem started at about 12:50 PM CT (17:50 UTC) on October 7, 2026. Other Stratus environments (CI, QA, RC) are not believed to be affected. Affected components Web (Degraded performance)

Status: Identified We have confirmed the cause. An authentication credential that Stratus uses to complete sign-in with our identity provider expired at about 12:45 PM CT (17:45 UTC) today. Sign-in attempts after that point fail at the final step. A replacement credential has been created. We are now applying it to the production servers and will confirm that sign-in works end to end. We'll post again once logins succeed. There is no data loss or security impact. Nothing is needed on your side. Next update: within 30 minutes, or as soon as sign-in is restored. Affected components Web (Degraded performance)

Status: Resolved Sign-in to Stratus (www.gtpstratus.com "www.gtpstratus.com": http://www.gtpstratus.com) is working normally again. Between about 12:50 PM CT and 1:30 PM CT (17:50 to 18:30 UTC) on October 7, 2026, users could not sign in. Cause: An authentication credential that Stratus uses to complete sign-in with our identity provider expired. After that point, every sign-in failed at the final step. Resolution: We replaced the expired credential and applied it to the production servers. Sign-in has succeeded since about 1:31 PM CT (18:31 UTC) and we have seen no further errors. Impact: All sign-ins to production Stratus failed during that window, about 40 minutes. Users with an existing active session were not logged out by this issue. There was no data loss, and we have found no security impact. Affected components Web (Operational)

Status: Investigating Still gathering info, reported by Crystal Ross Affected components Web (Partial outage)

Status: Resolved This incident has been resolved.

Status: Investigating Nearly every job queue on the platform (20+) has a large simultaneous backlog and DLQ buildup (e.g. loadstratusmodel 740 DLQ, modelversionrelease 573 DLQ, bigdatapublish 684 active/684 DLQ, jsontodocumentdb 340 active/242 DLQ, bim360publish 340 DLQ, loadstratusparts 370 DLQ). KEDA is reacting to all queues at once (ScaledJobs configured up to 100 concurrent replicas), driving 376 Job objects in stratusjobs and 1,277 pods cluster-wide. This volume of simultaneous object creation is overwhelming the AKS control plane (API server/etcd) — kubectl get pods took 6.4s. Control-plane strain is causing scheduling, kubelet pod-sync, and CNI IP allocation (CNS) to hang, so new pods (job pods and DaemonSets: retina-agent, CSI drivers, ama-metrics-node) sit in ContainerCreating/PodInitializing with no IP. Cluster autoscaler is compounding this by adding nodes (6+ new nodes across jobsazlnx03, lnxpdf2, primarylnx03, Windows pool in 15 min), each adding more control-plane load. Result: job pods miss the 1-min peek-lock, KEDA retries, messages exhaust delivery attempts into DLQ — happening simultaneously across queues, not just sendcutlist. Also observed: FailedCreatePodSandBox context deadline exceeded, worker exit from cluster DNS (10.210.0.10) timeout resolving MongoDB, and insufficient-CPU scheduling failures. Affected components Modules (Partial outage)

Status: Resolved This incident has been resolved.

Status: Investigating Impact: modules-dashboard.prd.gtpstratus.com intermittently returning 503 ("Micro-Frontend Unavailable" in the Stratus host) since ~18:35 UTC. Currently sustained, not self-resolving after 15+ min of monitoring. Root cause chain (working theory, not yet fully confirmed): 1. A burst of unrelated Stratus core job workload (stratusjobs/stratus/stratusapi namespaces — importbundle, sendcutlist, runpdfreport, etc.) triggered the cluster autoscaler to rapidly add nodes to the shared AKS node pools (aks-jobsazlnx03, aks-primarylnx03) that Modules also runs on. 2. modules-dashboard and one Valkey pod (valkey-0) got rescheduled onto newly-joined nodes during the churn. 3. Istio/CNI routing convergence for pods on brand-new nodes appears to be taking longer than expected, causing "no route to host" both on modules-dashboard's ingress path and on modules-backend's connections to Valkey (port 6379, PassthroughCluster — 15-20 failed connection attempts per 25s window, sustained). 4. Side effect: metrics-server was also disrupted mid-restart during the churn, causing unrelated HPA "FailedGetResourceMetric" noise across the cluster (istiod, istio-ingressgateway, App Config) — cosmetic, not part of the outage. Ruled out: this does NOT look like the Valkey Sentinel stale-ghost-entry bug fixed yesterday (PLAT-1879/PR #548) — that was permanently-dead entries never pruned; this is fresh/transient unreachability tied to active pod rescheduling. Separate, likely unrelated minor issue: modules-backend logged real 400s on assembly-tracking-status and package-tracking-status around 18:35, reported by a user updating a package's tracking status in the Viewer. Worth investigating on its own but doesn't explain the dashboard/Valkey connectivity failures. Status as of declare: still degraded — dashboard failing ~2/3 checks, Valkey still throwing ~20 connection errors per 25s window. Not converging on its own. Affected components Modules (Degraded performance)

Status: Resolved This incident has been resolved.