Dremio incident

Query failures in Dremio Cloud (US & EU)

Major Resolved View vendor source →

Dremio experienced a major incident on November 11, 2025 affecting Site Availability and Site Availability, lasting 9h 21m. The incident has been resolved; the full update timeline is below.

Started
Nov 11, 2025, 09:51 PM UTC
Resolved
Nov 12, 2025, 07:13 AM UTC
Duration
9h 21m
Detected by Pingoru
Nov 11, 2025, 09:51 PM UTC

Affected components

Site AvailabilitySite Availability

Update timeline

  1. investigating Nov 11, 2025, 09:51 PM UTC

    We are investigating issues with queries in Dremio Cloud (US & EU). We will provide an update within the next 30 minutes.

  2. investigating Nov 11, 2025, 10:15 PM UTC

    We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate the issue and will provide an update within the next 30 minutes.

  3. investigating Nov 11, 2025, 10:52 PM UTC

    We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate and narrow down the issue. We will provide an update within the next 30 minutes.

  4. investigating Nov 11, 2025, 11:22 PM UTC

    We are continuing to investigate the issue with queries in Dremio Cloud (US & EU). Engineering has narrowed down the issue and are working on multiple possible mitigations. We will provide an update within the next 45 minutes.

  5. investigating Nov 11, 2025, 11:58 PM UTC

    We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

  6. investigating Nov 12, 2025, 02:02 AM UTC

    We have narrowed down the issue impacting queries in Dremio Cloud (US & EU). Engineering is in the process of building and testing a fix to address the issue. We will provide an update within the next 45 minutes.

  7. identified Nov 12, 2025, 05:38 AM UTC

    We have identified the issue impacting queries in Dremio Cloud (US & EU) and are in the process of deploying a fix to address the issue. We will provide an update within the next 45 minutes.

  8. resolved Nov 12, 2025, 07:13 AM UTC

    Incident Summary We experienced a service disruption starting at 1:15 PM PST that resulted in query failures when using reflections in multiple regions. We sincerely apologize for the impact this had on your workflows and the inconvenience it may have caused. Incident Details An update to the permissions used by Dremio internal sources inadvertently restricted access to some of the folders in the project store, resulting in queries using reflections or accessing those folders to fail in some AWS projects under certain conditions. Our team began investigating the issue immediately and was able to identify the issue and implement a fix at 5 PM PST. Concurrently, we encountered cloud provider issues in our Azure deployment infrastructure which impacted our ability to roll out the fix to our production environments. Once we were able to resolve issue, we were able to validate and deploy a fix to fully restore the service at 10:35 PM PST. Our engineers are monitoring the system to ensure full stability. In addition, we are also implementing new alerts to detect such situation quickly in the future.