Dremio incident

Queries stuck in metadata retrieval in Dremio Cloud (EU)

Major Resolved View vendor source →

Dremio experienced a major incident on February 10, 2025, lasting 7h 53m. The incident has been resolved; the full update timeline is below.

Started
Feb 10, 2025, 10:02 AM UTC
Resolved
Feb 10, 2025, 05:55 PM UTC
Duration
7h 53m
Detected by Pingoru
Feb 10, 2025, 10:02 AM UTC

Update timeline

  1. investigating Feb 10, 2025, 10:02 AM UTC

    We are currently investigating an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.

  2. investigating Feb 10, 2025, 10:55 AM UTC

    We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.

  3. investigating Feb 10, 2025, 11:53 AM UTC

    We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.

  4. investigating Feb 10, 2025, 12:52 PM UTC

    We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.

  5. investigating Feb 10, 2025, 01:51 PM UTC

    We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.

  6. investigating Feb 10, 2025, 02:57 PM UTC

    We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.

  7. investigating Feb 10, 2025, 03:58 PM UTC

    We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We are experiencing issues with our upstream provider causing timeouts and are working to mitigate this issue. We will provide another update within 60 minutes.

  8. monitoring Feb 10, 2025, 05:15 PM UTC

    We have identified the issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage and applied a mitigation. We are seeing recovery of customer environments and will continue to monitor. We apologize for any inconvenience caused and will publish a post-mortem for this incident in the coming days.

  9. monitoring Feb 10, 2025, 05:55 PM UTC

    One of our core indexing services that powers both discovery of sources and metadata refreshes starting queueing up. We have mitigated the issue by throttling the service. Our initial analysis suggests that the failure may have been triggered by a recent change in our underlying infrastructure on Google. We are collaborating with our partners at Google to conduct a thorough Root Cause Analysis and identify the exact cause of the issue.

  10. resolved Feb 10, 2025, 05:55 PM UTC

    This incident has been resolved.