Dagster experienced a critical incident on November 2, 2022 affecting API, lasting 1h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 02, 2022, 07:44 PM UTC
The issue has been identified and a fix is being implemented.
- identified Nov 02, 2022, 07:44 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Nov 02, 2022, 08:29 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 02, 2022, 08:34 PM UTC
We upgraded our agent request queue backend at 2022-11-02 19:04 UTC. During upgrades, Dagster Cloud's backend services read messages from the old queue and write messages onto the new queue. Once the old queue is empty, Dagster Cloud's services begin reading messages only from the new queue. When rolling the new queue configuration out to Dagster Cloud's backend services, our scheduling services did not recognize the new configuration. These services continued to look only at the old queue. This means new agent requests were being written to the new queue, but our scheduler services - which control things like executing schedules, sensors, and launching runs - continued to only read from the old queue. Customers may need to manually retry runs that launched but were marked as failed during the outage window because Dagster Cloud never acknowledged the launch event. Schedules and sensors will resume automatically. To mitigate this in the future, when performing similar operations, we plan to use immutable configurations that will preclude services from continuing to run with an old configuration.