Qubole incident

Degraded performance issue while processing jobs on api.qubole.com

Minor Resolved View vendor source →

Qubole experienced a minor incident on February 11, 2022 affecting Command Processing, lasting 7h 14m. The incident has been resolved; the full update timeline is below.

Started
Feb 11, 2022, 08:52 AM UTC
Resolved
Feb 11, 2022, 04:07 PM UTC
Duration
7h 14m
Detected by Pingoru
Feb 11, 2022, 08:52 AM UTC

Affected components

Command Processing

Update timeline

  1. investigating Feb 11, 2022, 08:52 AM UTC

    api.qubole.com is currently seeing some degraded performance while processing commands. At this time issue appears to be partial, but DevOps is investigating on it.

  2. investigating Feb 11, 2022, 08:53 AM UTC

    We are continuing to investigate this issue.

  3. identified Feb 11, 2022, 11:04 AM UTC

    DevOps had identified that there is an issue with tunnel servers in api.qubole.com, and after creating new tunnels & doing sanity tests, faulty tunnels have been removed from the backend.

  4. monitoring Feb 11, 2022, 01:03 PM UTC

    Replacement of bad nodes is completed and now we are monitoring on it.

  5. resolved Feb 11, 2022, 04:07 PM UTC

    It seems that the tunnel server node issue is resolved as api.qubole.com is now working fine.