Dagster incident

Some Dagster Plus Serverless Runs Failing to Start

Major Resolved View vendor source →

Dagster experienced a major incident on May 14, 2024 affecting API, lasting 1h 22m. The incident has been resolved; the full update timeline is below.

Started
May 14, 2024, 06:53 PM UTC
Resolved
May 14, 2024, 08:16 PM UTC
Duration
1h 22m
Detected by Pingoru
May 14, 2024, 06:53 PM UTC

Affected components

API

Update timeline

  1. investigating May 14, 2024, 06:53 PM UTC

    Some Dagster Plus Serverless runs are failing to start with an "Agent detected that run worker failed." error. We are working with our cloud provider to investigate the root cause of the problem as well as adding automatic retries for the underlying error that is causing the runs to fail to start.

  2. investigating May 14, 2024, 07:34 PM UTC

    We are continuing to investigate the root cause of this issue with our cloud provider. We are rolling out a change to add additional automatic retries to runs that are affected by this issue.

  3. identified May 14, 2024, 07:57 PM UTC

    Our cloud provider has identified the root cause of the problem and is in the process of rolling back the change that caused it. We'll continue to provide updates as we work to resolve the issue.

  4. resolved May 14, 2024, 08:16 PM UTC

    The fix for this problem is now live.