Coalesce experienced a major incident on August 17, 2026 affecting Scheduler, lasting 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 17, 2026, 12:54 PM UTC
We have identified an issue with jobs executing and investigating the root cause. We will update this incident when resolved.
- investigating Aug 17, 2026, 12:56 PM UTC
We are investigating an issue affecting job runs for customers in our US region. A portion of scheduled and API-triggered job runs are failing to start. An affected run may remain in a Waiting state and then fail with the message "Run timed out. Restart the run, and if this issue persists contact support." Runs that do start are completing normally. Customers in our EU, Canada, and APAC regions are not affected. Our engineering team is actively investigating. We will post an update by 9:30 AM ET
- monitoring Aug 17, 2026, 01:17 PM UTC
We have identified the issue and pushed a change to resolve job processing. More details to follow once fully resolved.
- resolved Aug 17, 2026, 01:40 PM UTC
Between approximately 10:00 and 13:30 UTC today, some customers experienced delayed or stuck job runs. During a routine, provider-managed Kubernetes upgrade in one of our US regions, our cloud provider was unable to provision replacement compute capacity. Instead of aborting, the upgrade proceeded to take existing nodes out of rotation, leaving the region without capacity to start new jobs. We rolled back the upgrade, restoring full scheduling capacity, and jobs are processing normally. We were forced to cancel older jobs that were unable to recover and recommend customers re-run any effected operations. We are adjusting our upgrade configuration and maintenance windows to prevent this scenario from recurring.