Is Dagster down?

Impact: Dagster+ hybrid agents connecting to the US region via AWS PrivateLink experienced connection loss. Duration: ~30 minutes (17:35 – 18:05 UTC / 13:35 – 14:05 ET on 2026-08-29) Summary: As part of a planned infrastructure migration from our legacy nginx ingress to Istio, we rotated the network load balancers behind our PrivateLink endpoint service. This involved disconnecting the previous set of consumer VPC endpoints so that new connections would route through the Istio-backed path. During the ~30 minute window between endpoint disconnection and full re-establishment, hybrid agents connecting via PrivateLink were unable to reach the Dagster+ control plane and appeared as unhealthy in our health dashboard. Once new endpoints came online, agents automatically reconnected and returned to a healthy state without customer action. What was affected: Hybrid agents connecting to Dagster+ US region via AWS PrivateLink Job runs, code location updates, and metadata operations dependent on that connection were paused (queued locally) until the connection recovered What was NOT affected: The Dagster+ UI and API remained fully available Non-PrivateLink hybrid agents (using public network connectivity) Dagster+ Serverless (EU and US) Dagster+ EU hybrid customers Timeline (EDT): 12:35 — Existing PrivateLink consumer endpoints intentionally disconnected as part of the migration 12:40 – 12:55 — Agents in a disconnected state; new endpoint provisioning in progress 13:00 — Partial reconnection observed (~10% of affected agents) 13:05 — Full recovery to pre-migration baseline Root Cause: Planned migration step: swapping the network load balancers attached to the PrivateLink service. The gap in agent connectivity occurred while new endpoints were being provisioned on our side. The recovery took slightly longer than the pre-migration plan estimated because a subset of new endpoints hit a transient AWS API race condition on creation and required a retry.

**What happened** During a planned nginx→Istio migration on 2026-08-29 17:35 UTC, we disconnected existing PrivateLink consumer VPC endpoints as part of the cutover. Hybrid customer agents connecting via PrivateLink lost their connections and needed to reconnect via new endpoints on our side. Full recovery at 18:05 UTC, ~30 min. **What went well** * Migration itself completed cleanly; new endpoints healthy immediately after the retry apply. * Metric-based recovery detection \(`healthy_locations_external.ACTIVE`\) surfaced the impact and the recovery in real time. * No data loss; agents automatically re-attached with no customer action required. **What could have gone better** * We did not proactively communicate to affected customers ahead of the maintenance window. Realistically the class of affected customers is a small, known set \(PrivateLink hybrid\); we could have surfaced a maintenance banner in the product a few days in advance rather than a reactive Pylon blast when the disruption started. * Internal VPCE recreation hit a transient AWS API race \(private-DNS namespace not fully released before recreate\), prolonging the maintenance window.