Hatchet incident

[us-west-2] Elevated latency on us-west-2 cloud

Resolved after 5 hours 3 minutes, affecting API (cloud.hatchet.run) and API (Hatchet API), on August 22, 2026.

resolved Aug 22, 2026, 09:27 PM UTC

On August 22, 2026, starting around 12:00 UTC, one shared shard in the us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls caused lock contention and delayed task processing for tenants on that shard. Other shared shards in us-west-2, shared environments in other regions, and dedicated customer shards were not affected. Took steps to throttle the workload and reduce contention. Database load and scheduling latency on the affected shard returned to normal later the same afternoon. No data loss occurred.

Started
Aug 22, 2026, 04:24 PM UTC
Resolved
Aug 22, 2026, 09:27 PM UTC
Duration
5h 3m
Detected by Pingoru
Aug 22, 2026, 04:24 PM UTC

Affected components

API (cloud.hatchet.run)API (Hatchet API)

Update timeline

  1. investigating Aug 22, 2026, 04:24 PM UTC

    On August 22, 2026, starting around 12:00 UTC, the shared us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls on a shared shard caused lock contention and delayed task processing for tenants on that environment. We took action to throttle the work driving saturation and constraining scheduler capacity while the system recovered. Database load and scheduling latency returned to normal later the same afternoon. No data loss occurred.

  2. resolved Aug 22, 2026, 09:27 PM UTC

    On August 22, 2026, starting around 12:00 UTC, one shared shard in the us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls caused lock contention and delayed task processing for tenants on that shard. Other shared shards in us-west-2, shared environments in other regions, and dedicated customer shards were not affected. Took steps to throttle the workload and reduce contention. Database load and scheduling latency on the affected shard returned to normal later the same afternoon. No data loss occurred.

Your product runs on more than Hatchet

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →