Qubole incident

Degraded performance issue Airflow clusters on gcp.q

Minor Resolved View vendor source →

Qubole experienced a minor incident on February 21, 2023 affecting Cluster Operations, lasting 10h 22m. The incident has been resolved; the full update timeline is below.

Started
Feb 21, 2023, 04:47 PM UTC
Resolved
Feb 22, 2023, 03:10 AM UTC
Duration
10h 22m
Detected by Pingoru
Feb 21, 2023, 04:47 PM UTC

Affected components

Cluster Operations

Update timeline

  1. investigating Feb 21, 2023, 04:47 PM UTC

    For customers using Airflow, there is an issue with clusters that are running. All other aspects of Qubole operations are functioning normally. These Airflow clusters do not automatically terminate due to the nature of their function. Any currently running Airflow clusters are functioning normally. If an Airflow cluster is terminated or restarted for any reason then it will not come back up as the loading of Airflow will fail. We are currently working to resolve the issue.

  2. investigating Feb 21, 2023, 07:49 PM UTC

    We are continuing to work on a resolution.

  3. investigating Feb 21, 2023, 09:53 PM UTC

    We are continuing to work on a resolution.

  4. resolved Feb 22, 2023, 03:10 AM UTC

    We would inform you that the fix has been applied in the environment. We appreciate your patience.