Cloud 66 incident

Multiple Google Cloud services in the europe-west9 region are impacted

Minor Resolved View vendor source →

Cloud 66 experienced a minor incident on April 26, 2023 affecting Core Systems, lasting 16h 18m. The incident has been resolved; the full update timeline is below.

Started
Apr 26, 2023, 04:03 PM UTC
Resolved
Apr 27, 2023, 08:22 AM UTC
Duration
16h 18m
Detected by Pingoru
Apr 26, 2023, 04:03 PM UTC

Affected components

Core Systems

Update timeline

  1. investigating Apr 26, 2023, 03:36 PM UTC

    Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

  2. investigating Apr 26, 2023, 04:03 PM UTC

    Due to a major outage in Google Cloud (https://status.cloud.google.com/regional/europe) - Kubernetes installations may fail on installations on servers located in the same region, regardless of the cloud, due to Kubernetes images themselves being hosted on Google Cloud

  3. monitoring Apr 26, 2023, 04:37 PM UTC

    We are waiting on Google resolutions

  4. resolved Apr 27, 2023, 08:22 AM UTC

    Maestro installations are working as expected again