Dagster incident
Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error
Dagster experienced a major incident on September 6, 2023 affecting API, lasting 4h 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 06, 2023, 07:30 PM UTC
We have observed an increase in timeouts from our storage engine during run execution, resulting in some runs failing with an error like the following in the event log: dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server Error (Trace ID: XXX)', 'locations': [{'line': 22, 'column': 13}], 'path': ['eventLogs', 'getEventRecords']}] We are currently investigating the issue.
- identified Sep 06, 2023, 08:08 PM UTC
The issue has been identified and we are in the process of testing a fix.
- monitoring Sep 06, 2023, 10:39 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 06, 2023, 11:59 PM UTC
This incident has been resolved.