Dagster incident

Dagster Cloud sensors and runs failing

Major Resolved View vendor source →

Dagster experienced a major incident on January 27, 2023 affecting API, lasting 1h 36m. The incident has been resolved; the full update timeline is below.

Started
Jan 27, 2023, 07:14 PM UTC
Resolved
Jan 27, 2023, 08:50 PM UTC
Duration
1h 36m
Detected by Pingoru
Jan 27, 2023, 07:14 PM UTC

Affected components

API

Update timeline

  1. investigating Jan 27, 2023, 07:14 PM UTC

    We are investigating a failure at the graphql layer that is causing runs to fail.

  2. investigating Jan 27, 2023, 07:14 PM UTC

    We are continuing to investigate this issue.

  3. identified Jan 27, 2023, 07:40 PM UTC

    An error in the graphql layer is causing sensors and runs using the k8s_job_executor to fail

  4. monitoring Jan 27, 2023, 08:32 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Jan 27, 2023, 08:50 PM UTC

    This incident has been resolved.