Is Dagster down?

We've identified an issue with a background data-archival process in Dagster+ that may cause slower performance in some parts of the app. Runs, schedules, and sensors are not affected. A fix is being tested and we'll deploy it as soon as it's validated. We'll share further updates as progress is made.

We have deployed a partial fix to prevent the issue from worsening. We intend to continue our investigations and deploy a complete fix within the next 24 hours.

We have deployed a complete fix to our staging instance and are monitoring the situation. We intend to deploy a complete fix to our production instance within the next 24 hours.

We have deployed a complete fix to our production instance and confirm that this issue has been resolved.

Between 1:00pm and 1:24pm PT (20:00–20:24 UTC), Dagster+ had trouble communicating with agents. During this window, some sensor and schedule evaluations failed or timed out, and some runs may have started late. Service has recovered. Sensors and schedules are evaluating normally, and failed ticks will resume on their next evaluation. We're monitoring and investigating the root cause.

We are experiencing an issue with our production application cluster, impacting all Dagster+ cloud customers. We are currently investigating and working on a resolution.

We've identified the core issue, and are actively working to remediate. We are rolling back our deploy, and increasing the size of our database cluster to manage the increased load.

The rollback and upgraded database have completed and the system is operational.

This incident has been resolved.

# Dagster\+ UI timeouts and run failures On October 22, 2026, Dagster\+ customers in our US region experienced UI and API timeouts, failed runs, and delayed schedules and sensors for about two hours. The cause was a change in that day's release that made a frequently run database query much slower, which overloaded our primary database. We're sorry for the disruption, and especially for its effect on pipelines that depend on running on time. ## Customer Impact From approximately 16:40 UTC, customers saw slow or failing pages in the Dagster\+ UI, 504 errors and timeouts from the API, and failed runs. At the peak, roughly half as many runs completed successfully as normal. Schedules, sensors, and automation evaluations were delayed or skipped during the same period. Most symptoms subsided after we rolled back the release at about 17:40 UTC. Some customers continued to see intermittent errors until about 18:45 UTC, for the reason described below. ## What happened The October 22 release included a change to a query that runs every time a step in a run starts, which makes it one of the most frequently run queries in Dagster\+. The change was logically correct, but the production database chose a different, much less efficient query plan to execute it than the pre-production environments it was tested against, so each run of it took about 50 times longer than before. The extra load pushed the database to its CPU limit within a few minutes of the release. Requests across Dagster\+ then slowed or timed out, including those that start runs and record their progress. Because runs whose steps couldn't start retried, the load on the database increased further. ### Recovery We rolled the release back at about 17:35 UTC, which removed the slow query. At the same time, we moved the database to a server with twice the compute capacity. While the database was slow, our automatic scaling had increased the number of web server instances to its maximum, because the instances looked busy waiting for the database. When the database came back on its new server, all of those instances reconnected at once and exceeded the database's limit on concurrent connections. That caused the intermittent errors that continued until about 18:45 UTC, when we manually lowered the scaling limit and connections returned to normal. ## Prevention and follow-up We've completed the following: * Reverted the query change. * Doubled the compute capacity of the primary database. * Lowered the maximum number of web server instances to stay within the database's connection limit. * Routed web server database connections through a connection pooler, so that scaling up web servers can't exhaust the database's connections. We're also working on: * Redistributing database workloads off of this primary instance * Requiring that changes to database queries be tested against production-scale data before release, with automated review that flags query changes lacking that evidence. * Adding alerts on database CPU and connection usage, and checks during each release that pause it if database performance regresses. * Improving our process for updating customers during an incident.

Between 3:15 PM and 3:45 PM ET (12:15–12:45 PM PT, 19:15–19:45 UTC) on October 7, a backend issue delayed communication between Dagster+ and customer agents. During this window, some schedule, sensor and declarative automation ticks failed with "Unable to reach the user code server" errors, and some code location operations may have been slow or timed out. The issue was resolved by 3:45 PM ET (12:45 PM PT, 19:45 UTC), and ticks have run normally since. Schedule ticks that failed during this window may not have been retried automatically. If you're missing an expected run from that period, you can launch it manually from the Dagster+ UI. We're investigating the root cause and will take steps to prevent it from happening again.

We've deployed several fixes and performance optimizations that have resolved this issue, and should prevent it in the future.

# Root cause analysis On October 7 and 8, 2026, some Dagster\+ customers experienced failed run launches, timeouts, and delays in recording run events. During a release on October 7, the Redis service that carries requests between Dagster\+ and customer agents became overloaded. Our investigation also identified inefficient reuse of database connections in the service that records run events, which caused recurring delays over the following days. We’re sorry for the disruption and for the delay in communicating with customers during the incident. ## Customer Impact Beginning at approximately 19:15 UTC on October 7, some customers experienced requests timing out or runs failing to start. The initial disruption subsided around 19:39 UTC, but intermittent delays continued over the following days. During this period, Dagster\+ sometimes took longer to record updates from running jobs. Customers also reported missed runs after the initial disruption. These interruptions were particularly consequential for workflows that needed to run at precise intervals or collect information that could not be retrieved later. We applied several patches over the following days to reduce the impact, then resolved the recurring delays in recording run events with a fix on October 12. We confirmed the improvement through peak traffic on October 13, with no further latency warnings during that verification period. ## What happened ### October 7: Run launch failures During the Dagster\+ 1.13.26 rollout, the Redis service used for request storage reached its capacity. The service had limited headroom before the rollout, and the number of application instances restarting and reconnecting during the release increased load. As Redis became saturated, requests slowed or timed out. Application instances waiting on Redis consumed very little CPU; our automatic scaling interpreted that to be spare capacity and reduced the number of application instances, concentrating more work on the remaining instances. Redis recovered as its buffers cleared. Customer reports show that missed runs continued after request-storage Redis recovered. Engineering also observed elevated event-write latency during this period, although the available evidence does not establish a direct link to every reported missed run. ### October 8: Intermittent slowness The service that records run events repeatedly opened and closed database connections under load. Repeated recreation of Postgres connections was expensive and drove the latency spikes. Under high load, the service repeatedly created extra database connections and discarded them after use, spending time opening connections instead of processing run events. We made several more changes to reduce the impact of load on the cluster, and changed how the service reused database connections on October 12. This resolved the recurring delays, and we confirmed the improvement through peak traffic on October 13. ## Prevention and follow-up We increased the capacity of the request-storage Redis service, upgraded it to Valkey 8, increased the CPU limit on the event-writing service, and adjusted that service’s scaling behavior to reduce how quickly instances are removed. We are also evaluating limits to prevent Redis connection pools from growing without bounds, and will roll it out after testing in pre-production. Our planned follow-up work includes reviewing recurring capacity warnings, strengthening capacity checks during releases, and improving alerts for slow requests and growing backlogs. We also plan to connect customer support reports more closely with incident investigations and coordinate updates to affected customers.

Impact: Dagster+ hybrid agents connecting to the US region via AWS PrivateLink experienced connection loss. Duration: ~30 minutes (17:35 – 18:05 UTC / 13:35 – 14:05 ET on 2026-08-29) Summary: As part of a planned infrastructure migration from our legacy nginx ingress to Istio, we rotated the network load balancers behind our PrivateLink endpoint service. This involved disconnecting the previous set of consumer VPC endpoints so that new connections would route through the Istio-backed path. During the ~30 minute window between endpoint disconnection and full re-establishment, hybrid agents connecting via PrivateLink were unable to reach the Dagster+ control plane and appeared as unhealthy in our health dashboard. Once new endpoints came online, agents automatically reconnected and returned to a healthy state without customer action. What was affected: Hybrid agents connecting to Dagster+ US region via AWS PrivateLink Job runs, code location updates, and metadata operations dependent on that connection were paused (queued locally) until the connection recovered What was NOT affected: The Dagster+ UI and API remained fully available Non-PrivateLink hybrid agents (using public network connectivity) Dagster+ Serverless (EU and US) Dagster+ EU hybrid customers Timeline (EDT): 12:35 — Existing PrivateLink consumer endpoints intentionally disconnected as part of the migration 12:40 – 12:55 — Agents in a disconnected state; new endpoint provisioning in progress 13:00 — Partial reconnection observed (~10% of affected agents) 13:05 — Full recovery to pre-migration baseline Root Cause: Planned migration step: swapping the network load balancers attached to the PrivateLink service. The gap in agent connectivity occurred while new endpoints were being provisioned on our side. The recovery took slightly longer than the pre-migration plan estimated because a subset of new endpoints hit a transient AWS API race condition on creation and required a retry.

**What happened** During a planned network migration on 2026-08-29 17:35 UTC, we disconnected existing PrivateLink consumer VPC endpoints as part of the cutover. Hybrid customer agents connecting via PrivateLink lost their connections and were not able to reconnect until the new endpoints were provisioned. Full recovery at 18:05 UTC, ~30 min. **What went well** * No data loss; agents automatically re-attached with no customer action required. **What could have gone better** * Provisioning new endpoints took longer than expected. * We did not proactively communicate to affected customers ahead of the maintenance window. **Follow-ups** * We will update our standard procedures for planned maintenance to ensure communication of operations that may cause delays in customer pipelines.