Dagster incident

Issues with global op concurrency keys

Major Resolved View vendor source →

Dagster experienced a major incident on February 7, 2025 affecting API and Dagster Cloud UI, lasting 34m. The incident has been resolved; the full update timeline is below.

Started
Feb 07, 2025, 03:37 PM UTC
Resolved
Feb 07, 2025, 04:11 PM UTC
Duration
34m
Detected by Pingoru
Feb 07, 2025, 03:37 PM UTC

Affected components

APIDagster Cloud UI

Update timeline

  1. identified Feb 07, 2025, 03:37 PM UTC

    Some runs containing ops tagged with concurrency keys that have not been configured explicitly or with a default limit are getting blocked. Separately, there are some UI issues affecting the concurrency key settings page. We have identified fixes for these and are in the process of rolling them out.

  2. monitoring Feb 07, 2025, 03:56 PM UTC

    The fixes has been rolled out and previously blocked runs appear to be progressing. We are continuing to monitor for any issues. The previous blocked runs should progress with their step execution, but may not reach a terminal state and may need to be manually terminated.

  3. resolved Feb 07, 2025, 04:11 PM UTC

    This incident has been resolved.