Hatchet Outage History

On August 22, 2026, starting around 12:00 UTC, one shared shard in the us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls caused lock contention and delayed task processing for tenants on that shard. Other shared shards in us-west-2, shared environments in other regions, and dedicated customer shards were not affected. Took steps to throttle the workload and reduce contention. Database load and scheduling latency on the affected shard returned to normal later the same afternoon. No data loss occurred.

We saw elevated database load on our shared production environment beginning around 05:00 UTC, which caused some multi-step workflows to experience scheduling delays and increased API/dashboard latency for affected tenants. We reduced connection pressure on the shared database and latency has returned to normal. We’re continuing to monitor closely.

Between approximately 05:00–10:00 UTC on July 24, the shared cloud shard in our us-west-2 region experienced elevated database lock contention, which caused higher API and scheduling latency and some intermittent task-dispatch delays for tenants on that shard. This was isolated to a single shard in us-west-2 (cloud). Other regions and shards were not affected. We have no evidence of dropped workload, and all assignments were eventually completed. We are continuing to investigate and monitor.

We're continuing to monitor the fix. We've been seeing significant recovery since about 35 minutes ago (15:15 UTC). If you're seeing tasks that are stuck during execution, cancelling and replaying will likely get them moving again.

We have identified the source of the latency and have pushed initial configuration changes to mitigate. We will continue to monitor for degraded performance.

We’re back! Hatchet Cloud region us-west-2 should be up and running. Thanks for bearing with us.

Between 5:00 UTC and 16:25 UTC, we received reports of increased latency and error rates impacting some tenants in us-west-2. We have scaled capacity and all tenants are recovered.

This issue is now resolved and runs should be available again over the UI.

We have taken steps to mitigate affected DNS services and have fully restored all Hatchet services

On November 13, 2025, beginning at approximately 4:55 PM EST, our shared cloud environment experienced disruptions to queueing and monitoring for approximately one hour. The root cause was identified as a RabbitMQ memory saturation issue, which caused the message broker to enter an unrecoverable state. We have since restored full service, implemented monitoring improvements, and are working on several action items to prevent recurrence. This primarily impacted Free, Starter and Growth tier tenants, and did not impact tenants on the enterprise or dedicated plans.

We saw an increased number of 503 error rates between 1:40-1:50 UTC. We've deployed a fix and are monitoring now.

The Hatchet API is currently experiencing increased API error rates.