Is Windmill down?

Seeing degraded performance on Web App. Some users may experience high load times.

Web app degraded due to increased load. Additional resources have come online.

Windmill is experiencing high-load and elevated error rates. We have determined the root cause and are actively investigating a fix

Web App, Windy and MCP Unavailable Background Windmill generates Windy reports in the background to prepare content for our users. On October 12, 2026, a large number of these reports started at the same time. Our services connect to the database through a connection proxy. The proxy usually shares a small number of database connections between many requests. But a problem in how our services use their database connections stopped the proxy from sharing them. Each request held its own database connection, so the large report load opened thousands of connections, and each connection used memory on the database server. At 10:00 AM ET, the database server that handles reads ran out of memory and restarted two times. We added more read servers and the service recovered at 10:40 AM. At 11:44 AM, when the reports started to finish, the load moved to our primary database server, which handles writes. That server ran out of memory, and our database provider automatically moved the primary role to a standby server three times. Each move interrupted database connections for 1–2 minutes. During these periods, the Windmill web app, Windy and MCP were unavailable or slow, and many requests failed. Some review chats were sent later than planned. Timeline 10/12 @ 10:00 AM ET - Report generation starts. The read database server runs out of memory. 10/12 @ 10:09 AM ET - Web app becomes unavailable. Status page updated at 10:10 AM. 10/12 @ 10:10 AM ET - Cause linked to report generation. 10/12 @ 10:16 AM ET - Read database server restarts and recovers, but some requests continue to fail. 10/12 @ 10:25 AM ET - Read database server runs out of memory a second time. 10/12 @ 10:33 AM ET - Additional read servers added. 10/12 @ 10:40 AM ET - Error rate drops to 0%. Incident resolved and monitoring continues. 10/12 @ 11:25 AM ET - Read servers slow down again under load. More read servers added. 10/12 @ 11:44 AM ET - Primary database server runs out of memory. The primary role moves to a standby server. 10/12 @ 12:00 PM ET - Reports finish, load moves to write database. 10/12 @ 12:04 PM ET - Primary database server runs out of memory a second time. Status page updated at 12:07 PM. 10/12 @ 12:42 PM ET - Primary database server runs out of memory a third time. 10/12 @ 1:05 PM ET - Query improvements deployed. Database memory recovers and errors stop. 10/12 @ 1:28 PM ET - Incident resolved and monitoring continues. Remediation We understand the impact this downtime had; our customers rely on Windmill for critical work and we apologize greatly for interrupting it. Internally, we are implementing changes to prevent similar incidents from occurring in the future. This includes: Lowering the maximum number of connections each database server can hold Continuing to optimize our database queries for agent-driven workloads Upgrading our performance review systems to spread report generation over a longer period of time

We are experiencing intermittent issues connecting to our MCP server. We are investigating the situation.

Background Early this week, Windmill rolled out enhanced monitoring for the core infrastructure which processes all requests coming into all our services (Windy, web app, MCP, and integrations). We noticed a small number (>0.01%) of requests were failing due to a networking misconfiguration in how our services talk to each other. Although there was no customer impact since these failing requests get retried, we proactively rolled out a fix the evening of 9/30/2026. Windmill runs its services in a high-availability configuration, to ensure our services are always available. To maintain high-availability during upgrades, we use rolling deploys where old versions of services stay online until the new version is ready. Once the new version is ready, the old version enters a draining state where it will finish servicing all pending requests before it goes offline. The configuration change we rolled out on 9/30/2026 caused a latent bug in how we handle draining to block the old versions from shutting down. We balance load evenly across all instances, meaning the 50% of requests going to the stuck old version would fail. Timeline 9/30 @ 11:52 PM EST - Fix deployed for networking misconfiguration, exposing latent bug in service shutdown 10/1 @ 7:00 AM EST - A small number of MCP servers get shut down during load balancing, but fail to drain, causing minor degradation in MCP service. 10/1 @ 8:30 AM EST - Error rates increase, reported internally and incident declared. 10/1 @ 9:00 AM EST - Root cause identified. 10/1 @ 9:46 AM EST - Fix is identified, applied, and roll out begins. 10/1 @ 9:52 AM EST - Roll out reaches our API servers, error rates begin to rise as the old version fails to drain properly. 10/1 @ 10:01 AM EST - First user-facing errors accessing app.gowindmill.com reported and incident escalated. 10/1 @ 10:09 AM EST - Roll out of new API servers completes, but old servers continue to fail to drain. 50% error rate observed. 10/1 @ 10:21 AM EST - Infrastructure vendor involved to forcefully shutdown the erroring API servers. 10/1 @ 10:25 AM EST - Force shutdown triggered by vendor, error rates remain elevated 10/1 @ 10:30 AM EST - Identified another service was blocking the force shutdown, restart on the other service triggered 10/1 @ 10:36 AM EST - Restart completes, all the erroring servers terminate, error rate drops to 0%. 10/1 @ 10:43 AM EST - Continued monitoring reveals no further errors, incident resolved. Remediation We understand the impact this downtime had - our customers rely on Windmill for critical work and we apologize greatly for interrupting it. Internally, we are implementing changes to prevent similar incidents from occurring in the future. This includes: Enhanced testing in staging prior to promotion to production Comprehensive audits for similar latent risks and remediation Continuing to improve monitoring and shorten incident response time More clear error pages For continuous updates on Windmill’s real-time status, we encourage customers to subscribe to status page here: https://status.gowindmill.com/

Windmill is under high load and experiencing degraded performance

Services have been scaled up and are now stable.

We are investigating delays from our queuing provider affecting some Windmill background workflows. Some jobs may start or finish later than expected. We are working to restore normal operation and will share another update when we have more information

We are continuing to monitor this but the syncing system is healthy now and is working as normal