Dagster incident

Reports of jobs not starting

Major Resolved View vendor source →

Dagster experienced a major incident on May 3, 2023 affecting API, lasting 30m. The incident has been resolved; the full update timeline is below.

Started
May 03, 2023, 10:58 PM UTC
Resolved
May 03, 2023, 11:29 PM UTC
Duration
30m
Detected by Pingoru
May 03, 2023, 10:58 PM UTC

Affected components

API

Update timeline

  1. investigating May 03, 2023, 10:58 PM UTC

    We have reverted a recent change and believe this should resolve the issue. We will provide an update after further investigation.

  2. investigating May 03, 2023, 11:12 PM UTC

    The issue is confirmed as resolved, though we are continuing to investigate the issue.

  3. resolved May 03, 2023, 11:29 PM UTC

    This incident has been resolved. An operation to upgrade the nodes of our cluster overlapped with our weekly release. Interaction between the two operations caused the Dagster daemons that control run queuing, alerting, schedules, and sensors to be temporarily unavailable.