Dremio incident

Intermittent engine failure in Dremio Cloud (US)

Notice Resolved View vendor source →

Dremio experienced a notice incident on December 17, 2024 affecting Dremio Cloud - US, lasting 2h 5m. The incident has been resolved; the full update timeline is below.

Started
Dec 17, 2024, 04:56 PM UTC
Resolved
Dec 17, 2024, 07:01 PM UTC
Duration
2h 5m
Detected by Pingoru
Dec 17, 2024, 04:56 PM UTC

Affected components

Dremio Cloud - US

Update timeline

  1. investigating Dec 17, 2024, 04:56 PM UTC

    We are currently investigating an issue impacting engines within Dremio Cloud (US) that are causing them to shutdown while queries are in progress. We will provide another update within 30 minutes.

  2. identified Dec 17, 2024, 05:27 PM UTC

    We have identified the issue impacting engine stability within Dremio Cloud (US). We are working to mitigate this issue. We will provide another update within the next 30 minutes.

  3. identified Dec 17, 2024, 06:01 PM UTC

    We have identified the issue impacting engine stability within Dremio Cloud (US). We are continuing to work to mitigate this issue. We will provide another update within the next 30 minutes.

  4. identified Dec 17, 2024, 06:42 PM UTC

    We are in the process of mitigating the issue. More updates will be provided in 30 minutes.

  5. identified Dec 17, 2024, 06:52 PM UTC

    We have identified the issue impacting engine stability within Dremio Cloud (US). We have applied changes to our infrastructure and are working on recovering services. We will provide another update within the next 30 minutes.

  6. identified Dec 17, 2024, 06:54 PM UTC

    We are continuing to work on a fix for this issue.

  7. monitoring Dec 17, 2024, 06:58 PM UTC

    We have mitigated the issue impacting engine stability within Dremio Cloud (US) and are continuing to monitor the recovery of customer engines. We apologize for any inconvenience caused.

  8. resolved Dec 17, 2024, 07:01 PM UTC

    We have mitigated the issue impacting engine stability within Dremio Cloud (US) and have seen successful recovery of customer engines. We apologize for any inconvenience caused and will publish a post-mortem for this incident in the coming days.