Hatchet incident
[us-west-2] Elevated latency on us-west-2 cloud
Hatchet experienced a minor incident on August 22, 2026 affecting API (cloud.hatchet.run) and API (Hatchet API), lasting 5h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 22, 2026, 04:24 PM UTC
On August 22, 2026, starting around 12:00 UTC, the shared us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls on a shared shard caused lock contention and delayed task processing for tenants on that environment. We took action to throttle the work driving saturation and constraining scheduler capacity while the system recovered. Database load and scheduling latency returned to normal later the same afternoon. No data loss occurred.
- resolved Aug 22, 2026, 09:27 PM UTC
On August 22, 2026, starting around 12:00 UTC, one shared shard in the us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls caused lock contention and delayed task processing for tenants on that shard. Other shared shards in us-west-2, shared environments in other regions, and dedicated customer shards were not affected. Took steps to throttle the workload and reduce contention. Database load and scheduling latency on the affected shard returned to normal later the same afternoon. No data loss occurred.