Google Outage History

Started Oct 25, 2026, 12:00 PM UTC· resolved Nov 02, 2026, 05:54 PM UTC

1. monitoring Oct 26, 2026, 10:01 AM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: Starting October 25, 2026 at around 05:00 PT, some requests to the Geocoding API (all platforms), Places API Legacy & New (all platforms) and Address Validation API may return incomplete or incorrectly formatted address information for street-level results. Affected responses may be missing address components such as city, region, postal code or country. They may also return street names in an unexpected language or with incorrect address formatting, and may be flagged as partial matches. Results may vary between requests. Unexpectedly empty responses may be returned for street searches and queries. Our engineering team has identified the root-cause and is working on a mitigation. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

2. monitoring Oct 26, 2026, 10:58 AM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We are experiencing an issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS beginning at Sunday, 2026-10-25 05:00 PDT. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2026-10-26 05:00 PDT with current details. We apologize to all who are affected by the disruption. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

3. monitoring Oct 26, 2026, 11:53 AM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: Mitigation work is currently underway by our engineering team. We will provide more information by Monday, 2026-10-26 08:00 PDT. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

4. monitoring Oct 26, 2026, 02:57 PM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: Our Engineering Team has identified the root cause of the issue and mitigation work is underway. The issue was triggered by a recent map data update that omitted certain regional and administrative relationships for street addresses. Engineering has stopped further updates from this pipeline and is rolling back the change while regenerating corrected datasets in parallel. Because the data restoration requires multi-stage processing and verification across our serving systems, the mitigation is expected to take several hours to fully complete. We will provide more information by Monday, 2026-10-26 10:00 PDT. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

5. monitoring Oct 26, 2026, 04:34 PM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: Our Engineering Team has identified the root cause of the issue and mitigation work is underway. The issue was triggered by a recent map data update that omitted certain regional and administrative relationships for street addresses. Engineering has stopped further updates from this pipeline and is rolling back the change while regenerating corrected datasets in parallel. Because the data restoration requires multi-stage processing and verification across our serving systems, the mitigation is expected to take several hours to fully complete. We will provide more information by Monday, 2026-10-26 11:00 PDT. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

6. monitoring Oct 26, 2026, 06:07 PM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: Our Engineering Team has identified the root cause of the corrupted data issue and mitigation work to roll back the change and regenerate corrected datasets is in progress. The data restoration requires multi-stage processing and verification across our serving systems. The mitigation is expected to take several hours to fully complete. We will provide more information by Monday, 2026-10-26 13:30 PDT. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

7. monitoring Oct 26, 2026, 08:09 PM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Monday, 2026-10-26 14:40 PDT. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

8. monitoring Oct 26, 2026, 09:56 PM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: The mitigation work to roll back to the corrected dataset in our data respositories was completed. Those changes will take additional time now to be picked up by our indexing and serving systems. While we are already seeing a drop in degraded results where response has missing address components, the fix will take additional time to fully propagate to all services. We will provide more information by Monday, 2026-10-26 16:45 PDT. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

9. monitoring Oct 27, 2026, 12:13 AM UTC

   Summary: Incomplete and/or incorrectly localized addresses for some street results Description: The resolution of the issue is still underway. We're seeing rate of degraded results such as missing address components and wrong localization return to pre-outage levels for Geocoding and Places API. We are continuing to track the progress of the data propagation across services and will provide an update by Monday, 2026-10-26 19:00:00 PDT. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

10. monitoring Oct 27, 2026, 01:51 AM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is partially resolved. We do not have an ETA for full resolution at this point. We will provide an update by Monday, 2026-10-26 23:00 PDT with current details. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

11. monitoring Oct 27, 2026, 04:53 AM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is now mostly resolved. Our engineering team continues to closely monitor the services to address any remaining edge cases. We do not have an ETA for full resolution at this point. We will provide an update by Tuesday, 2026-10-27 11:00 PDT with current details. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

12. monitoring Oct 27, 2026, 02:27 PM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is mostly resolved globally. However, we still observe a small number of impacted features, primarily in the geographical regions of Taiwan, Brazil, Vietnam, Colombia and Singapore. We do not have an ETA for full resolution at this point. We will provide an update by Tuesday, 2026-10-27 10:00 PDT with current details. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

13. monitoring Oct 27, 2026, 04:58 PM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is resolved. Recovery has completed across previously affected regions, including Taiwan, Brazil, Vietnam, Colombia, and Singapore. Our engineering team is verifying data restoration for a small remaining subset of road features before closing this incident. We do not have an ETA for full resolution at this point. We will provide an update by Tuesday, 2026-10-27 11:00 PDT with current details. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

14. monitoring Oct 27, 2026, 07:07 PM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is resolved for the vast majority of traffic. Recovery has completed across previously affected regions, including Taiwan, Brazil, Vietnam, Colombia, and Singapore. Our engineering team is completing final data restoration and verification for a small remaining subset of road features before closing this incident. We do not have an ETA for full resolution at this point. We will provide an update by Tuesday, 2026-10-27 14:00 PDT with current details. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time. Diagnosis: Missing address components, missing or wrong localization, incorrect formatting, missing results Workaround: None at this time.

15. monitoring Oct 28, 2026, 09:10 PM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: Our engineering team has completed the rollback of the remaining affected road features, and updated map data has propagated across our production serving systems. We are performing final verification of our automated data pipelines before resolving this incident. We will provide an update by Thursday, 2026-10-29 12:00 PDT. Diagnosis: Customers using Directions API, Routes API, Distance Matrix API, and Navigation SDK may have experienced degraded route quality, suboptimal detours, or missing road segments in multiple regions, including Taiwan, Brazil, Vietnam, Colombia, and Singapore. Workaround: None at this time.

16. monitoring Oct 29, 2026, 07:27 PM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is partially resolved. While we believe user impact has been resolved, internal processes and procedures remain paused while Google applies the fixes that will allow their resumption. We do not have an ETA for full resolution at this point. We will provide an update by Thursday, 2026-10-29 19:00 PDT with current details. Diagnosis: Customers using Directions API, Routes API, Distance Matrix API, and Navigation SDK may have experienced degraded route quality, suboptimal detours, or missing road segments in multiple regions, including Taiwan, Brazil, Vietnam, Colombia, and Singapore. Workaround: None at this time.

17. monitoring Oct 30, 2026, 03:40 AM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is resolved. Our engineering team is completing final data restoration and verification before closing this incident. We will provide an update by Friday, 2026-10-30 15:00 PDT with current details. Diagnosis: No ongoing symptoms detected. Workaround: None at this time.

18. monitoring Oct 30, 2026, 09:44 PM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is resolved. Our engineering team is completing final data restoration and verification before closing this incident. We will provide an update by Friday, 2026-10-30 16:30 PDT with current details. Diagnosis: No ongoing symptoms detected. Workaround: None at this time.

19. monitoring Oct 30, 2026, 11:21 PM UTC

    Summary: Incomplete and/or incorrectly localized addresses for some street results Description: We believe the issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS is resolved. Our engineering team is completing final data restoration and verification before closing this incident. We will provide an update by Monday, 2026-11-02 10:30 PDT with current details. Diagnosis: No ongoing symptoms detected. Workaround: None at this time.

20. resolved Nov 02, 2026, 05:54 PM UTC

    The issue with Address Validation API, Geocoding API, Places API, Places API (New), Places Library, Maps JavaScript API, Places SDK for Android, Places SDK for iOS has been resolved for all affected users as of Friday, 2026-10-30 15:45 PDT. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue.

Read the full incident report →

Started Oct 23, 2026, 02:40 AM UTC· resolved Oct 23, 2026, 04:42 AM UTC

1. resolved Oct 25, 2026, 02:39 PM UTC

   Google Engineering mitigated this issue with a rollback in the reverse geocoding serving stack.

2. resolved Nov 02, 2026, 08:04 PM UTC

   An Incident Report for the service degradation affecting the Geocoding API on October 22, 2026 is now available. For a detailed analysis of the event, including the root cause and our prevention measures, please view the published Incident Report on the Public Issue Tracker here: https://issuetracker.google.com/issues/576897533

Read the full incident report →

Started Oct 22, 2026, 10:54 PM UTC· resolved Oct 24, 2026, 08:00 AM UTC

1. investigating Oct 22, 2026, 10:56 PM UTC

   The issue has been identified and mitigation is in process.

Read the full incident report →

Started Oct 21, 2026, 01:05 PM UTC· resolved Oct 22, 2026, 03:30 AM UTC

1. investigating Oct 21, 2026, 02:39 PM UTC

   Users may be experiencing intermittent 500's and elevated latencies.

2. resolved Oct 22, 2026, 07:19 PM UTC

   We've successfully mitigated the network overloads on RTDB's backend that were causing network delays and intermittent 500's to REST requests.

Read the full incident report →

Started Oct 20, 2026, 06:00 AM UTC· resolved Oct 20, 2026, 12:08 PM UTC

1. investigating Oct 20, 2026, 08:35 AM UTC

   Description: We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2026-10-20 02:30 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region. Workaround: None for now

2. investigating Oct 20, 2026, 09:24 AM UTC

   Description: We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2026-10-20 04:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region. Note: Looker Studio is not impacted by this outage. Workaround: If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).

3. investigating Oct 20, 2026, 11:00 AM UTC

   Description: We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2026-10-20 05:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region. Note: Looker Studio is not impacted by this outage. Workaround: If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).

4. investigating Oct 20, 2026, 11:12 AM UTC

   Description: We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Due to a power event we had suffered an outage within europe-west2-b zone. The power event is mitigated. Partial mitigation to infrastructure services will be observed within the next 30 minutes and the remaining services will be mitigated there after. We will provide an update by Tuesday, 2026-10-20 05:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Google Cloud services are unavailable in europe-west2-b zone; some services in europe-west2 region may experience elevated errors and intermittent network issues. Workaround: If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).

5. investigating Oct 20, 2026, 11:58 AM UTC

   Description: We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. A power event caused an outage within europe-west2-b zone. The power event is mitigated. We are seeing the recovery of services and are prioritizing the recovery of existing workloads.Customers may not be able to yet create new workloads. Full recovery of existing workloads should be completed, tentatively by Tuesday, 2026-10-20 06:00. We will provide an update by Tuesday, 2026-10-20 06:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Google Cloud services are unavailable in europe-west2-b zone; some services in europe-west2 region may experience elevated errors and intermittent network issues. Workaround: If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).

6. investigating Oct 20, 2026, 12:44 PM UTC

   Description: We are seeing the recovery of services for most workloads in europe-west2 as of Tuesday, 2026-10-20 05:10 PDT. We will provide an update by Tuesday, 2026-10-20 07:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Google Cloud services are unavailable in europe-west2-b zone; some services in europe-west2 region may experience elevated errors and intermittent network issues. Workaround: If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).

7. investigating Oct 20, 2026, 02:03 PM UTC

   Description: The issue with impacted GCP products has been mitigated for most of the customers and it is believed to be affecting currently a very small number of customers and our Engineering Team is working on it. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. No further updates will be provided here. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we're working on resolving the issue. Symptoms: Google Cloud services were unavailable in europe-west2-b zone; some services in europe-west2 region might experienced elevated errors and intermittent network issues. Workaround: None as the issue has been mitigated.

8. resolved Oct 22, 2026, 08:42 AM UTC

   Preliminary Incident Report We recognize that Google Cloud services are critical to your organization, and we sincerely apologize for the disruption this incident may have caused to your business. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https\://cloud.google.com/support (https://cloud.google.com/support). Date/Time of the Issue Incident Start:19 Oct 2026 23:00 US/Pacific / 20 Oct 2026 07:00 BST Incident End: 20 Oct 2026 05:10 US/Pacific / 20 Oct 2026 13:10 BST Duration: 6 hours, 10 minutes Summary On Monday, 19 October 2026 (Tuesday, 20 October BST), multiple Google Cloud Platform services in the europe-west2 region experienced service disruptions, elevated error rates, Application Programming Interface (API) timeouts, and processing latency for a duration of 6 hours and 10 minutes. The disruption was triggered by an electrical power interruption at the datacenter facility hosting availability zone europe-west2-b, which temporarily took physical host servers, network routing hardware, and storage systems offline. This subsequently caused secondary failover and queuing delays across unimpacted regional infrastructure. Preliminary Root Cause The incident was triggered when a power grid disruption occurred at the datacenter facility hosting availability zone europe-west2-b. To maintain continuous power during the disruption, automatic transfer to facility backup power systems was triggered immediately. The facility load was transferred to the Uninterruptible Power Supply (UPS) battery banks during the start up sequence of on-site emergency diesel generators. The Programmable Logic Controller (PLC) automation system (both primary and backup PLCs) failed to initiate the logical automatic start/transfer operations of the generators, preventing backup generators from assuming the load. When UPS battery reserves were depleted, physical host servers, network switching fabric, and local storage arrays dedicated to europe-west2-b lost power and went offline. The sequence of events leading to customer impact was as follows: * At 22:53 US/Pacific (20 Oct, 06:53 BST), an electrical power disruption occurred at the datacenter facility hosting availability zone europe-west2-b, and a PLC (Programmable Logic Controller) startup fault prevented facility emergency generators from assuming the electrical load. Facility load continued to be supported by UPS battery banks until the batteries were depleted, starting at 23:00 US/Pacific (20 Oct, 07:00 BST). * When the physical host servers, local storage infrastructure, and network aggregation hardware in europe-west2-b lost power, it caused instances, database replicas, and worker nodes in that zone to become immediately unreachable. * Regional traffic-routing mechanisms and client applications initiated automated failover toward available infrastructure in adjacent availability zones within the region. * Cross-zone remote procedure calls from regional service control planes directed to metadata and catalog components in the offline zone encountered timeouts, causing initial job submission stalls and API latency spikes. * The sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots and processing capacity in other availability zones, creating elevated admission queues, delayed database failover completions, and intermittent 500 and 503 API errors across the region. Google engineers have begun a full root cause analysis (RCA) and will provide additional information once it is available. Remediation Google engineers were alerted to the disruption via automated monitoring systems and probers at 23:08 US/Pacific on 19 Oct (20 Oct, 07:08 BST), which detected elevated API error ratios, job submission timeouts, and loss of infrastructure reachability in europe-west2-b. Engineering teams immediately mobilized to isolate the affected infrastructure and commence recovery: * Traffic Isolation: Google engineers isolated the offline availability zone (europe-west2-b) by adjusting regional service routing away from the affected infrastructure, preventing cross-zone client requests from timing out against offline endpoints. * Workload Reprioritization and Resource Safeguards: Engineers adjusted scheduler allocations and capacity limits across unimpacted infrastructure to prioritize active customer reservations, reduce resource contention on execution slots, and relieve queue backlogs for regional workloads. * Power Grid Restoration: On-site facility engineers conducted graceful restoration of a stable utility power feed once utility supply was back, within 1 hour. Following this, they focused and completed a reset and tested the emergency backup generator automation controllers to restore facility power. * Progressive Infrastructure Re-activation: Google engineers systematically restarted core Google Cloud infrastructure and services. * Controlled Traffic Re-admission: Once system health and storage integrity were verified, engineers executed gradual, controlled traffic re-admission back to zone europe-west2-b allowing accumulated job backlogs to clear smoothly. By 05:10 US/Pacific on Tuesday, 20 Oct(13:10 BST), service availability had recovered for the vast majority of customer workloads, and resolution for most of the customers across Google Cloud Platform services in the region was confirmed. Following the recovery of a small tail of remaining isolated workloads, the closure notification for the incident was posted by 07:03 US/Pacific (15:03 BST). Description of Impact Between Monday, 19 Oct 2026, 23:00 US/Pacific (Tuesday, 20 Oct 2026, 07:00 BST) and Tuesday, 20 October 2026, 05:10 US/Pacific (13:10 BST), customers utilizing Google Cloud Platform services in the europe-west2 region experienced service disruptions across two consecutive phases, lasting a total duration of 6 hours, 10 minutes. During the initial phase from 23:00 to 04:29 US/Pacific (07:00 to 12:29 BST) (duration 5 hours, 29 minutes), customers experienced elevated error rates, API timeouts, and processing delays until most Google Cloud Platform services recovered. Following platform service restoration, some customer workloads running on Compute Engine instances experienced a longer recovery tail. By 04:46 US/Pacific (12:46 BST) new Virtual Machine creation resumed and by 05:10 US/Pacific (13:10 BST) over 99% of Compute Engine Virtual Machines were in a healthy state. Any remaining customers that were experiencing Virtual Machine problems were directed to file a customer support issue as per normal procedure. While the physical power outage was isolated strictly to datacenter facilities in europe-west2-b, some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact. The primary causes were 1) instantaneous demand surges on unimpacted compute capacity and 2) synchronous control-plane validation dependencies across zonal boundaries. Affected products included: BigQuery, Looker, Google Compute Engine, Google Kubernetes Engine, Regional Persistent Disk, and Cloud SQL. These services experienced unexpected degradation and failover issues driven by sudden compute capacity exhaustion from surging demand, and control-plane API timeouts waiting on validations from the offline zone. Workloads and resources hosted entirely within europe-west2-b became unreachable. Data storage services with failover capability such as Cloud SQL, AlloyDB, Memorystore for Redis, and Cloud Bigtable failed over into nearby unimpacted zones during the europe-west2-b outage. Multi-zonal and regional services that spanned across the europe-west2 region experienced elevated latency, transient API errors (such as HTTP 500 and 503 errors), and task admission queuing as failover traffic concentrated onto unimpacted regional resources and cross-zone calls to offline components timed out. Storage write operations in the region experienced a transient error rate during the initial failover period, and regional query admission queues temporarily peaked before rapidly subsiding as capacity re-balanced. The following Google Cloud Platform services experienced impact: * AlloyDB for PostgreSQL * Apigee * BigQuery Data Transfer Service * Cloud Data Fusion * Cloud Filestore * Cloud Firestore * Cloud Load Balancing * Cloud Memorystore * Cloud NAT * Cloud Run * Cloud Spanner * Cloud Workstations * Datastream * Dialogflow CX * Google App Engine * Google BigQuery * Google Cloud Bigtable * Google Cloud Dataflow * Google Cloud Dataproc * Google Cloud NetApp Volumes * Google Cloud Pub/Sub * Google Cloud SQL * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Google SecOps (Security Information and Event Management [SIEM] and Security Orchestration, Automation, and Response [SOAR]) * Looker (Google Cloud core) * Looker Studio * Managed Airflow (Google Cloud Composer) * Managed Service for Apache Kafka * Memorystore for Memcached * Memorystore for Redis * Memorystore for Redis Cluster * Memorystore for Valkey * Network Connectivity Center * Persistent Disk * Private Service Connect * Speech-to-Text * Virtual Private Cloud (VPC) * VMWare engine

9. resolved Oct 27, 2026, 03:13 PM UTC

   Incident Report Summary On Monday, 19 Oct 2026 (Tuesday, 20 Oct BST), a data center facility that hosts europe-west2-b, experienced an external electrical power disruption. This resulted in service unavailability for workloads located in this zone. As traffic was redirected to alternative cloud regions or unaffected zones within europe-west2 cloud region, these workloads experienced transient latency and elevated errors affecting services across europe-west2. The primary impact was mitigated after 6 hours 8 minutes. Root Cause Facility architecture and background: The data center facility in the UK region hosts Cloud clusters serving the europe-west2-b zone, which form part of the multi-zonal europe-west2 Cloud region. The site relies on a 2N utility power architecture—meaning the facility has complete duplicate power paths for every critical load—backed by a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC). Primary utility power is supplied to the facility via two independent feeders (A and B), supplemented by a standby utility connection configured for manual campus-wide transfer during maintenance operations or emergency events. The two high-voltage primary feeders terminate at separate primary intake switchboards capable of closed-loop operation if either board is taken offline. Switching between the primary and standby utility providers utilizes open transitions and key interlocks to prevent parallel connections across sources. Static Uninterruptible Power Supply (UPS) battery systems provide 5 minutes of autonomous runtime. This bridge power ensures uninterrupted operation while the PLC detects utility undervoltage events, initiates diesel generator startup, synchronizes the generators, and transfers the site load. Prior to the utility disruption, the facility had been placed in a temporary single-feed (1N) bridged maintenance posture on the primary utility. Circuit breaker feeder B was physically opened and key-interlocked (Locked Out Tagged Out) to allow for corrective repairs following issues identified during a quarterly generator load test on 14 Oct 2026. Consequently, primary utility feed A was supporting the total critical load while repairs were conducted on the B feeder pathway. Root cause: On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 06:53 BST), the primary utility provider experienced an outage on primary utility feed A, caused by an external underground cable fault on the utility network outside the data center facility. When the primary utility feed A was lost, the facility’s generator plant failed to receive a signal to automatically start and take critical load due to a design voltage-sensing and misconfiguration in the PLCs. The facility's static UPS battery banks carried the critical load as designed until their battery reserves depleted resulting in a complete power loss across data halls. The investigation of the PLC configuration identified the root cause as a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. Instead, the PLC undervoltage sensing inputs were wired upstream (line side) of the isolated utility feed B breaker. The PLC was programmed to verify whether voltage was present on the feeder before commanding a generator start. As the utility feed B remained energized upstream, both PLCs sensed live upstream voltage on the disconnected path. As a result, the PLC logic indicated that a healthy utility source was available, even though the primary and secondary distribution feeds were de-energized, and did not issue a start command to the generator plant.; The onsite facilities team attempted to execute a remote load transfer via the PLC; however, the system inhibited the action. Recognizing that the UPS battery reserves would be exhausted before further remote operations could be resolved, the team transferred the facility load to the standby utility source, restoring the critical load that was lost once batteries depleted during this process. Once utility power was restored via the standby utility source, and UPS batteries started recharging, the onsite facility teams and OEM vendors reconfigured and validated PLC functionality on both undervoltage and breaker status. After the power loss was initially detected, Google engineers began work to mitigate failover failures and prepare service recovery. Upon restoration of the utility source, UPS and generator capabilities, they began the process to re-enable services in europe-west2-b. Remediation and Prevention The sequence of events leading to customer impact was as follows: * 19 Oct 22:53 US/Pacific (20 Oct BST 06:53 BST) - An electrical power disruption occurred at the data center facility hosting availability zone europe-west2-b, and a PLC (Programmable Logic Controller) startup fault prevented facility emergency generators from assuming the electrical load. Facility load continued to be supported by UPS battery banks until the batteries were depleted. * 19 Oct 23:00 US/Pacific (20 Oct 07:00 BST) Impact start - As battery backups were exhausted, physical host servers, local storage infrastructure, and network aggregation hardware in availability zone europe-west2-b lost power, causing instances, database replicas, and worker nodes in that zone to become immediately unreachable. * 19 Oct 23:04 US/Pacific (20 Oct 07:04 BST) - Google engineers were alerted to the disruption via automated monitoring systems and probers, which detected elevated API error ratios, job submission timeouts, and loss of infrastructure reachability in zone europe-west2-b. Engineering teams immediately mobilized to isolate the affected infrastructure and commence recovery. * 19 Oct 23:09 US/Pacific (20 Oct 07:09 BST) - Regional traffic-routing mechanisms and client applications initiated automated failover toward available infrastructure in adjacent availability zones within the region. * Cross-zone remote procedure calls from regional service control planes directed to metadata and catalog components in the offline zone encountered timeouts, causing initial job submission stalls and API latency spikes. * The sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots and processing capacity in other availability zones, creating elevated admission queues, delayed database failover completions, and intermittent 500 and 503 API errors across the region. * Traffic Isolation: Google engineers isolated the offline availability zone (europe-west2-b) by adjusting regional service routing away from the affected infrastructure, preventing cross-zone client requests from timing out against offline endpoints. * Workload Reprioritization and Resource Safeguards: Engineers adjusted scheduler allocations and capacity limits across unimpacted infrastructure to prioritize active customer reservations, reduce resource contention on execution slots, and relieve queue backlogs for regional workloads. * 19 Oct 23:54 US/Pacific (20 Oct 07:54 BST) Utility Power Restoration - Power supply from the standby utility source was restored and power restoration to the data hall began. * 20 Oct 00:20 US/Pacific (08:20 BST) Datahall Power Restoration - Utility power was re-established to the datahalls. Mains utility power normalized; power feed to data hall verified. Following this, the site operations and engineering teams focused on and completed a reset and tested the emergency backup generator automation controllers to restore facility power. * 20 Oct 00:40 US/Pacific (08:40 BST) - Clusters began to recover, cluster manager systems started rescheduling compute jobs. * 20 Oct 00:45 US/Pacific (08:45 BST) - Network devices were back online. * 20 Oct 00:50 US/Pacific (08:50 BST) - Due to the immediacy of power loss, storage devices with unclean shutdowns began recovery. Responders recognized this and chose not to restore services that would otherwise make this worse. Because of the large volume of storage devices involved, this process required a substantial period to finish prior to restoring service to customers. * 20 Oct 03:13 US/Pacific (11:06 BST) - Power redundancy was verified and the storage infrastructure was sufficiently restored to prepare the initiation of controlled traffic re-admission. The remaining storage systems continued recovery operations. * 20 Oct 03:45 US/Pacific (11:45 BST) Controlled Traffic Re-admission - Once system health and storage integrity were verified, engineers executed gradual, controlled traffic re-admission back to zone europe-west2-b allowing accumulated job backlogs to clear smoothly. * 20 Oct 04:11 US/Pacific (12:11 BST) - Google Cloud Platform services and APIs were made fully available to customers. Some Google Compute Engine virtual machines were still not yet available. * 20 Oct 04:29 US/Pacific (12:29 BST) - Regional services and backend compute management systems were fully undrained, marking the end of core platform-level disruption. * 20 Oct 05:08 US/Pacific (13:08 BST) Majority Workload Recovery - GCE virtual machines were 99.5% operational and healthy, and creating new VMs was fully supported. Service availability was restored for most customer workloads, with resolution confirmed for the majority of users across Google Cloud Platform services in the region. Following the recovery of a small tail of remaining isolated workloads, the closure notification for the incident was posted by 07:03 US/Pacific (15:03 BST). Google is committed to preventing a repeat of this issue in the future and is completing the following actions: Data center action items: * Restoration of normal utility configuration (both A & B feeds) * Utility provider to locate the damaged cable fault, using low-frequency radar penetration testing procedures. [Completed 26 Oct] * Utility provider to send on-site engineers to repair the cable fault and restore the impacted primary utility feed A source. [Estimated Completion Date 30 Oct] * Onsite Operations teams with a controlled method of procedure (MOP) to transfer the facility to its primary utility source (feeds A and B) from standby utility feed through coordinated switching with the utility provider [Estimated Completion Date 31 Oct] * Generator confidence testing * Full generator load transfer tests with utility power cutover of the active utility feed. Test automatic startup of generators through PLC signals within the backup battery timer and detailed review of backup procedures [Completed on 20 Oct and 26 Oct (full critical load transfer test including updated EOPs)] * Full generator load transfer tests in coordination with the primary utility source feeds (A and B) after restoration work completed by the primary utility provider. [Estimated Completion Date 31 Oct] * PLC configuration fix * Reconfigured and validated the PLC after the outage on 20 Oct and then again on 26 Oct [Completed 26 Oct] * Google engineers to work in collaboration with the facility vendor teams to perform a comprehensive PLC code review and system resiliency to multiple failure scenarios. [Estimated Completion Date (prior to generator transfer test) estimated 31 Oct] * Validate PLC telemetry once primary utility feed A and B lineup is restored. [Estimated Completion Date 31 Oct] * Validate global PLC configurations at all global sites with similar topologies and PLC manufacturers, to ensure confidence in automatic generator starts. [Estimated Completion Date 15 Nov] * Emergency Operating Procedures (EOP) documentation update: * Comprehensive update to the EOP for site’s response to unexpected loss of active utility source. Update includes - manual steps to attempt to restore active utility feed, manual restart of generators and manual transfer to other utility sources within the UPS battery run time. [Completed 26 Oct] * Ensure updated EOP is practiced twice per shift by on-site facility team. [Completed 24 Oct and checked with Google DC Operations daily] * Strengthen change management: * Freeze planned maintenance activities at other sites with similar topologies, and implement elevated approvals and reviews for data center configuration changes which are outside their standard designs. [Estimated Completion Date 30 Oct] Cloud service action items: * Communications Enhancement: Bolster notification speed by developing an automated alert-to-communication mechanism for location-impacting incidents. * Platform Testing: Expand platform-wide testing outcomes for infrastructure-level events. Detailed Description of Impact Between Monday, 19 Oct 2026, 23:00 US/Pacific (Tuesday, 20 Oct 2026, 07:00 BST) and Tuesday, 20 Oct 2026, 05:08 US/Pacific (13:08 BST), customers utilizing Google Cloud Platform services in the europe-west2 region experienced service disruptions across two consecutive phases, lasting a total duration of 6 hours, 08 minutes. During the initial phase from 23:00 to 04:29 US/Pacific (07:00 to 12:29 BST) (duration 5 hours, 29 minutes), customers experienced unavailability of zone-scoped resources in products, including: * Compute Engine - VM instances, zonal MIGs, unmanaged instance groups, zonal NEGs, sole-tenant node groups, zonal reservations, target instances, zonal API operations. * Persistent Disk / Hyperdisk - Zonal PD, zonal Hyperdisk, Hyperdisk storage pools, instant snapshots. * GKE - Zonal clusters (control plane), node pool nodes in -b, PVs on zonal disks. * Cloud SQL - Non-HA instances, read replicas in -b. * AlloyDB - Basic (non-HA) primaries in -b. * Bigtable - Single-cluster instances in -b. * Other products with affected zonal resources: Cloud Batch, Cloud Build, Cloud Storage, Cloud Workstations, Composer, Dataproc, Filestore, Managed Lustre, Migrate to VMs, NetApp Volumes, Pub/Sub Lite, Vertex AI Workbench, VMware Engine. The unavailability of those resources scoped to a zone that undergoes disruption is working within the design of a zonally-scoped system. Requests made to these resulted either in a timeout or error code during the disruption. Any workloads and resources hosted entirely within europe-west2-b became unreachable due to this event. Google Compute Engine's zonal resources such as VMs require customers to provision failover or cross-zone recovery resources if needed, and to set up a recovery mechanism that matches their needs. Some failovers were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism. Following platform service restoration, some customer workloads running on Compute Engine instances as well as products that rely on Compute Engine experienced a longer recovery. By 04:46 US/Pacific (12:46 BST) new Virtual Machine creation resumed and by 05:08 US/Pacific (13:08 BST) over 99% of Compute Engine Virtual Machines were in a healthy state. Any remaining customers that were experiencing Virtual Machine problems were directed to file a customer support case as per normal procedure. While the physical power outage was isolated strictly to data center facilities in zone europe-west2-b, some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact. The primary causes of impact outside of zone europe-west2-b were: 1) Instantaneous demand surges on neighboring zones (a and c) and 2) Control-plane dependencies across zonal boundaries Affected products included: AlloyDB, Apigee Infra and Apigee Edge, BigQuery, Cloud Dataflow, Cloud Pub/Sub, Cloud SQL, Google Compute Engine, Google Kubernetes Engine, Looker, Regional Persistent Disk, Vertex AI Prediction, and Cloud Speech-to-Text. These services experienced unexpected degradation and failover issues driven by sudden compute capacity exhaustion from surging demand, and control-plane API timeouts. Customers would have experienced the inability to create, update, or delete some regional resources during this period. This was not a full regional control-plane outage, but certainly a degraded state. Multi-zonal and regional services that spanned across the europe-west2 region experienced elevated latency, transient API errors (such as HTTP 500 and 503 errors), and task admission queuing as failover traffic concentrated onto unimpacted regional resources and cross-zone calls to offline components timed out. Storage write operations in the region experienced a transient error rate during the initial failover period, and regional query admission queues temporarily peaked before rapidly subsiding as capacity re-balanced. We did not identify any impact of the outage outside of the europe-west2 region. The following Google Cloud Platform services experienced impact within europe-west2: * AlloyDB for PostgreSQL * Apigee * BigQuery Data Transfer Service * Cloud Data Fusion * Cloud Filestore * Cloud Firestore * Cloud Load Balancing * Cloud Memorystore * Cloud NAT * Cloud Run * Cloud Spanner * Cloud Workstations * Datastream * Dialogflow CX * Google App Engine * Google BigQuery * Google Cloud Bigtable * Google Cloud Dataflow * Google Cloud Dataproc * Google Cloud NetApp Volumes * Google Cloud Pub/Sub * Google Cloud SQL * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Google SecOps (Security Information and Event Management [SIEM] and Security Orchestration, Automation, and Response [SOAR]) * Looker (Google Cloud core) * Looker Studio * Managed Airflow (Google Cloud Composer) * Managed Service for Apache Kafka * Memorystore for Memcached * Memorystore for Redis * Memorystore for Redis Cluster * Memorystore for Valkey * Network Connectivity Center * Persistent Disk * Private Service Connect * Speech-to-Text * Virtual Private Cloud (VPC) * VMWare engine ----------

10. resolved Nov 04, 2026, 09:15 PM UTC

    Incident Report Summary On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 2026, 06:53 BST), a data center facility that hosts europe-west2-b, a zone within the europe-west2 cloud region, experienced an external electrical power disruption. This resulted in service unavailability for workloads located in this zone. The subsequent failover of these workloads to unaffected zones and alternate regions caused transient latency and elevated error rates across the broader europe-west2 region. The resolution of the main impact was completed within 6 hours and 8 minutes. Root Cause Facility architecture and background: The data center facility in the UK region hosts Cloud clusters serving the europe-west2-b zone, which form part of the multi-zonal europe-west2 Cloud region. The site relies on a 2N utility power architecture—where the facility has complete duplicate power paths for every critical load—backed by a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC). Primary utility power is supplied to the facility via two independent feeders (A and B), supplemented by a standby utility connection configured for manual campus-wide transfer during maintenance operations or emergency events. The two high-voltage primary feeders (A and B) terminate at separate primary intake switchboards capable of closed-loop operation if either board is taken offline. Switching between the primary and standby utility providers utilizes open transitions and key interlocks to prevent parallel connections across sources. Uninterruptible Power Supply (UPS) battery systems provide approximately five minutes of autonomous runtime. This bridge power ensures uninterrupted operation while the PLC detects utility undervoltage events, initiates diesel generator startup, synchronizes the generators, and transfers the site load. Prior to the utility disruption, the facility had been placed in a temporary single-feed (1N) bridged maintenance posture on the primary utility. Circuit breaker feeder B was physically opened to allow for corrective repairs following issues identified during a quarterly generator load test on 14 Oct 2026. Consequently, primary utility feed A was supporting the total critical load while the feed B breaker remained in an open position to allow repairs to be conducted on the B feeder pathway. Root cause: On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 2026, 06:53 BST), the primary utility provider experienced an outage on primary utility feed A, caused by an external underground cable fault on the utility network outside the data center facility. Following the loss of primary utility feed A, a voltage-sensing design flaw and PLC misconfiguration prevented the facility’s generator plant from receiving an automatic start signal to assume the critical load. Consequently, static UPS battery banks sustained the critical load as intended until their reserves were used up, leading to a loss of power across the data halls. The investigation of the PLC configuration identified the root cause as a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. Instead, the PLC undervoltage sensing inputs were wired upstream (line side) of the isolated utility feed breakers. The PLC was programmed to verify whether voltage was present on the line side of the feeder before commanding a generator start. At the time of the power event on 19 Oct 2026, the Utility feed B remained energized upstream while its incomer feed was in open position. Consequently, the active and standby master PLCs, which were running identical control logic, sensed live voltage from feed B through shared upstream sensors on the disconnected path. As a result, the PLC logic indicated that a healthy utility source was available, even though the primary and secondary distribution feeds were de-energized, and did not issue a start command to the generator plant. The onsite facilities team attempted to execute a remote load transfer via the PLC; however, the system inhibited the action due to the live upstream voltage reading on feed B. Recognizing that the UPS battery reserves would be used up before further remote operations could be resolved, the team transferred the facility load to the standby utility source, restoring the critical load that was lost once batteries depleted during this process. Once utility power was restored via the standby utility source, and UPS batteries started recharging, the onsite facility teams and OEM vendors reconfigured and validated PLC functionality on both undervoltage and breaker status. Following the initial detection of the power outage, Google engineering teams initiated efforts to address failover issues and lay the groundwork for service restoration. Once the utility feed, UPS systems, and generator capabilities were successfully restored, engineers initiated the recovery procedure to bring europe-west2-b services back online. Remediation and Prevention The sequence of events leading to customer impact was as follows: * 19 Oct 22:53 US/Pacific (20 Oct 2026, 06:53 BST) - An electrical power disruption occurred at the data center facility hosting availability zone europe-west2-b, and a PLC (Programmable Logic Controller) startup fault prevented facility emergency generators from assuming the electrical load. Facility load continued to be supported by UPS battery banks until the batteries were depleted. * 19 Oct 23:00 US/Pacific (20 Oct 2026, 07:00 BST) - Impact start - As battery backups were used up, physical host servers, local storage infrastructure, and network aggregation hardware in availability zone europe-west2-b lost power, causing instances, database replicas, and worker nodes in that zone to become immediately unreachable. * 19 Oct 23:04 US/Pacific (20 Oct 2026, 07:04 BST) - Google engineers were alerted to the disruption via automated monitoring systems and probers, which detected elevated API error ratios, job submission timeouts, and loss of infrastructure reachability in zone europe-west2-b. Engineering teams immediately mobilized to isolate the affected infrastructure and commence recovery. * 19 Oct 23:09 US/Pacific (20 Oct 2026, 07:09 BST) - Regional traffic-routing mechanisms and client applications initiated automated failover toward available infrastructure in adjacent availability zones within the region. * Cross-zone remote procedure calls from regional service control planes directed to metadata and catalog components in the offline zone encountered timeouts, causing initial job submission stalls and API latency spikes. * The sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots and processing capacity in other availability zones, creating elevated admission queues, delayed database failover completions, and intermittent 500 and 503 API errors across the region. * Traffic Isolation: Google engineers isolated the offline availability zone (europe-west2-b) by adjusting regional service routing away from the affected infrastructure, preventing cross-zone client requests from timing out against offline endpoints. * Workload Reprioritization and Resource Safeguards: Engineers adjusted scheduler allocations and capacity limits across unimpacted infrastructure to prioritize active customer reservations, reduce resource contention on execution slots, and relieve queue backlogs for regional workloads. * 19 Oct 23:54 US/Pacific (20 Oct 2026, 07:54 BST) - Utility Power Restoration- Power supply from the standby utility source was restored and power restoration to the data hall began. * 20 Oct 00:20 US/Pacific (20 Oct 2026, 08:20 BST) - Datahall Power Restoration- Utility power was re-established to the datahalls. Mains utility power normalized; power feed to data hall verified. Following this, the site operations and engineering teams focused on and completed a reset and tested the emergency backup generator automation controllers to restore facility power. * 20 Oct 00:40 US/Pacific (20 Oct 2026, 08:40 BST)- Clusters began to recover, cluster manager systems started rescheduling compute jobs. * 20 Oct 00:45 US/Pacific (20 Oct 2026, 08:45 BST) - Network devices were back online. * 20 Oct 00:50 US/Pacific (20 Oct 2026, 08:50 BST) - Due to the immediacy of power loss, storage devices with unclean shutdowns began recovery. Responders recognized this and chose not to restore services that would otherwise make this worse. Because of the large volume of storage devices involved, this process required a substantial period to finish prior to restoring service to customers. * 20 Oct 03:13 US/Pacific (20 Oct 2026, 11:13 BST)- Power redundancy was verified and the storage infrastructure was sufficiently restored to prepare the initiation of controlled traffic re-admission. The remaining storage systems continued recovery operations. * 20 Oct 03:45 US/Pacific (20 Oct 2026, 11:45 BST)- Once system health and storage integrity were verified, engineers executed gradual, controlled traffic re-admission back to zone europe-west2-b allowing accumulated job backlogs to clear smoothly. * 20 Oct 04:11 US/Pacific (20 Oct 2026, 12:11 BST)- Google Cloud Platform services and APIs were made fully available to customers. Some Google Compute Engine virtual machines were still not yet available. * 20 Oct 04:29 US/Pacific (20 Oct 2026, 12:29 BST) - Regional services and backend compute management systems were fully undrained, marking the end of core platform-level disruption. * 20 Oct 05:08 US/Pacific (20 Oct 2026, 13:08 BST) - Majority Workload Recovery- GCE virtual machines were 99.5% operational and healthy, and creating new VMs was fully supported. Service availability was restored for most customer workloads, with resolution confirmed for the majority of users across Google Cloud Platform services in the region. Following the recovery of a small tail of remaining isolated workloads, the closure notification for the incident was posted by 07:03 US/Pacific (20 Oct 2026, 15:03 BST). Google is committed to preventing a repeat of this issue in the future and is completing the following actions: Data center action items: * Restoration of normal utility configuration (both A & B feeds) * Utility provider to locate the damaged cable fault, using low-frequency radar penetration testing procedures. [Completed 26 Oct] * Utility provider to repair the cable, test and restore the impacted primary utility feed A source. [Estimated Completion Date 13 Nov] * Onsite Operations teams with a controlled method of procedure (MOP) to transfer the facility to its primary utility source (feeds A and B) from standby utility feed through coordinated switching with the utility provider [Estimated Completion Date 17 Nov] * Generator confidence testing * Full generator load transfer tests with utility power cutover of the active utility feed. Test automatic startup of generators through PLC signals within the backup battery timer and detailed review of backup procedures [Completed on 20 Oct and 26 Oct (full critical load transfer test including updated EOPs)] * Full generator load transfer tests in coordination with the primary utility source feeds (A and B) after restoration work completed by the primary utility provider. [Estimated Completion Date 17 Nov] * PLC configuration fix * Reconfigured and validated the PLC after the outage on 20 Oct and then again on 26 Oct [Completed 26 Oct] * Google engineers to work in collaboration with the facility vendor teams to perform a comprehensive PLC code review and system resiliency to multiple failure scenarios. [Estimated Completion Date (prior to generator transfer test) 06 Nov] * Validate PLC telemetry once primary utility feed A and B lineup is restored. [Estimated Completion Date 17 Nov] * Validate global PLC configurations at sites with similar topologies and PLC manufacturers [Estimated Completion Date 15 Nov] * Emergency Operating Procedures (EOP) documentation update: * Comprehensive update to the EOP for site’s response to unexpected loss of active utility source. Update includes - manual steps to attempt to restore active utility feed, manual restart of generators and manual transfer to other utility sources within the UPS battery run time. [Completed 26 Oct] * Ensure updated EOP is practiced twice per shift by on-site facility team. [Completed 24 Oct and checked with Google DC Operations daily] * Detect and prevent future PLC misconfigurations * Google to ensure PLC code reviews are fully conducted and validated during facility commissioning and Integrated System Tests (ISTs) prior to Facility Ready (FR). A second Google team will then validate this within a quarter post-FR. [Estimated Completion Date 18 Dec] * Strengthen change management: * Freeze planned maintenance activities at other sites with similar topologies, and implement elevated approvals and reviews for data center configuration changes which are outside their standard designs. [Completed Date 30 Oct] Cloud service action items: * Communications Enhancement: Bolster notification speed by developing an automated alert-to-communication mechanism for location-impacting incidents. * Platform Testing: Expand platform-wide testing outcomes for infrastructure-level events. Detailed Description of Impact Between Monday, 19 Oct 2026, 23:00 US/Pacific (Tuesday, 20 Oct 2026, 07:00 BST) and Tuesday, 20 Oct 2026, 05:08 US/Pacific (Tuesday, 20 Oct 2026, 13:08 BST), customers utilizing Google Cloud Platform services in the europe-west2 region experienced service disruptions across two consecutive phases, lasting a total duration of 6 hours, 08 minutes. During the initial phase from 23:00 US/Pacific (07:00 BST) to 04:29 US/Pacific (12:29 BST) (duration 5 hours, 29 minutes), customers experienced unavailability of zone-scoped resources in products, including: * Compute Engine - VM instances, zonal MIGs, unmanaged instance groups, zonal NEGs, sole-tenant node groups, zonal reservations, target instances, zonal API operations. * Persistent Disk / Hyperdisk - Zonal PD, zonal Hyperdisk, Hyperdisk storage pools, instant snapshots. * GKE - Zonal clusters (control plane), node pool nodes in -b, PVs on zonal disks. * Cloud SQL - Non-HA instances, read replicas in -b. * AlloyDB - Basic (non-HA) primaries in -b. * Bigtable - Single-cluster instances in -b. * Other products with affected zonal resources: Cloud Batch, Cloud Build, Cloud Storage, Cloud Workstations, Composer, Dataproc, Filestore, Managed Lustre, Migrate to VMs, NetApp Volumes, Pub/Sub Lite, Vertex AI Workbench, VMware Engine. The unavailability of those resources scoped to a zone that undergoes disruption is working within the design of a zonally-scoped system. Requests made to these resulted either in a timeout or error code during the disruption. Workloads and resources hosted entirely within europe-west2-b became unreachable due to this event. While the physical power outage was isolated strictly to data center facilities in zone europe-west2-b, some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact. Google Cloud designs regional services to be robust to zone failures, but in this case there were a set of unexpected failures which caused impact outside of the zonal failure domain. We are committed to learning from this incident and preventing zonal events from impacting broader regional operations. The primary causes of impact outside of zone europe-west2-b were: 1) Instantaneous demand surges on neighboring zones (a and c) and 2) Control-plane dependencies across zonal boundaries See below for more detailed descriptions of impact and next steps for each impacted product. Google Compute Engine and Persistent Disk Impact: Upon power loss in europe-west2-b at 22:53 US/Pacific (06:53 BST), all Compute Engine VMs and zonal Persistent Disk and Hyperdisk volumes in the zone became unavailable. Google Compute Engine's zonal resources, such as VMs, require customers to provision cross-zone failover and recovery mechanisms tailored to their needs. During this incident, however, some of these failover attempts were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism. Once data hall power was restored at 00:20 US/Pacific (08:20 BST), engineers worked to confirm storage integrity before restoring compute services. Following platform service restoration, some customer workloads running on Compute Engine instances as well as products that rely on Compute Engine experienced a longer recovery due to action taken by responders in order to prevent data loss. Once storage stabilized, engineers brought VMs and zonal disks in the zone back online between 04:01 US/Pacific (12:01 BST) and 04:29 US/Pacific (12:29 BST). By 04:46 US/Pacific (12:46 BST) new VM creation resumed, and by 05:08 US/Pacific (13:08 BST) over 99% of Compute Engine VM were in a healthy state. (The remaining VMs were restarted after their disks recovered). Any remaining customers that were experiencing VM problems were directed to file a customer support case as per normal procedure. Next steps: Google Compute Engine and Persistent Disk engineering teams are conducting a comprehensive postmortem review of this event along with Google data center and infrastructure teams. The follow-up actions focus on preventing a recurrence and on making our clusters recover from an unplanned power loss more quickly and safely. We are also expanding our regular disaster-recovery testing for full cluster power-loss scenarios to validate end-to-end Compute Engine recovery. The zonal failure in a facility within europe-west2-b exhibited sequencing atypical of previous zonal outages. We are investigating additional signals to incorporate into our failure detection capabilities for Regional Persistent Disk, so that should this sequence recur, Regional Persistent Disk can differentiate this case from the more typical zonal outage scenarios. We exercise caution with all changes to detection mechanisms to ensure we do not regress our capabilities to detect and react to more typical outages. Cloud SQL Impact: Upon power loss, Cloud SQL detected failures and immediately initiated automated failovers for all regional HA instances. The failover was complete for regional instances that did not have Regional Persistent Disk failures (see above), mitigating their impact right away. A majority of the remaining instances recovered by 20 Oct 2026, 01:00 US/Pacific (09:00 BST) once the Regional Persistent Disk recovered, shortly after europe-west2-b recovery. A small number of remaining instances were fully recovered by 20 Oct 2026, 11:00 US/Pacific (19:00 BST). Next steps: Our engineering team is implementing improvements across our database and storage infrastructure to ensure that High Availability instances using Regional Persistent Disks recover more rapidly during unplanned outages. Google Kubernetes Engine Impact: The power loss in europe-west2-b caused total zonal failure, autoscaling stalls, API error spikes, and regional pod unavailability across GKE clusters. Total incident duration for GKE was 7 hours and 34 minutes, starting at 23:02 US/Pacific (07:02 BST), reaching initial mitigation at 05:22 US/Pacific (13:22 BST), and achieving final resolution at 06:37 US/Pacific (14:37 BST). Zonal clusters in europe-west2-b experienced 100% unavailability between 23:05 US/Pacific (07:05 BST) and 04:30 US/Pacific (12:30 BST), with full restoration by 06:00 US/Pacific (14:00 BST). Regional clusters lost worker nodes in europe-west2-b, causing a spike in regional unavailable pods. Worker nodes scaled up in neighboring zones europe-west2-a and europe-west2-c. The scale ups in surviving zones were impacted by available capacity. Autoscaling was slowed down due to increased load and failures of underlying APIs, such as Google Compute Engine (see above). Regional API operations suffered increased failure rates for cluster creation, deletion, updates and node pool operations due to reduced capacity and increased load on underlying APIs (see above). Google Cloud Dataflow Impact: Dataflow mostly behaved as expected for a zonal failure: new batch jobs were automatically routed away from europe-west2-b within about one minute, and all workloads (streaming and batch) recovered with no data loss once power returned. Jobs running in the affected zone also resumed responding to user cancellation requests once the zone came back up. Additionally, we identified and have now fixed an issue that caused increased streaming job start latency for about two hours. Google engineering is actively working to resolve a zone-selection issue that incorrectly routed a small number of new jobs to the unavailable zone. Looker Impact: The power event interrupted Looker's connectivity to GKE and CloudSQL, which Looker uses for its backend compute and metadata database resources, respectively. About 45% of instances became unresponsive as Looker's messaging middle tier was unable to recover and reestablish database connectivity due to degraded GKE and Cloud SQL availability. A manual rolling restart of messaging infrastructure across all instances brought unavailable Looker instances down to about 15% within about two hours of onset. Subsequently, regional control-plane issues in GKE pushed instance unavailability back to about 35% before settling near 20% for several hours. Cloud SQL failover degradation extended the delay as well. Availability returned to pre-incident baseline by 05:20 US/Pacific (13:20 BST) as the GKE outage was mitigated; the incident was declared mitigated at 07:44 US/Pacific (15:44 BST). Next Steps: Engineering teams are working to expand zonal outage telemetry and improve Looker's messaging middleware failovers to shorten recovery times during zonal unavailability. AlloyDB Impact: Customer impact for AlloyDB began at 23:07 US/Pacific (07:07 BST). Regional primary AlloyDB instances experienced up to a 95-minute delay in starting automated failovers due to delayed failover notifications. ~4% of these instances experienced a stalled failover due to Persistent Disk issues that required manual recovery. Additionally, zonal primary instances, zonal secondary instances, and single-node read pool instances encountered downtime directly aligned with the zone's power loss. Over 99% of affected AlloyDB instances in the region were restored to a healthy state by 06:16 US/Pacific (14:16 BST), with the final zonal primary AlloyDB instances recovering by 08:28 US/Pacific (16:28 BST). Next steps: AlloyDB engineering team will be updating their monitoring system to detect zonal outage reliably and send the failover notification. The AlloyDB team is also partnering with the Persistent Disk team to ensure that high availability instances which leverage regional disk will recover faster during unplanned downtime events. BigQuery Impact: BigQuery experienced a service disruption in the europe-west2 region between 23:03 US/Pacific (07:03 BST) and 04:12 US/Pacific (12:12 BST) (approximately 5 hours and 9 minutes). During this incident, affected customers experienced significant increases in job latencies and high query queuing, resulting in delayed job execution. Customers also experienced intermittent 503 UNAVAILABLE errors on Job API calls (InsertJob, GetQueryResult). Similarly, customers received errors when calling AppendRows on buffered-type streams also after the incident onset until the streams were re-created. Finally, customers experienced reduced processing efficiency and elevated job failure rates across affected projects. The majority of the impact was confined to customers assigned to the impacted zone and was caused by an autoscaling dependency in a critical data plane job, which was unable to scale up during the failover. Some customers not assigned to the impacted zone saw elevated (1%) error rates on JobAPI calls due to overly broad concurrency throttling in the Job API component. Next steps: Engineering teams have updated job provisioning to eliminate the dependency on autoscaling and are enhancing isolation mechanisms for the job server to ensure that future zonal disruptions are contained. Memorystore Redis Impact: During the power disruption, Basic Tier (non-HA) instances located in europe-west2-b experienced full unavailability, while Standard Tier (HA) instances initiated automated failovers or operated in a degraded state with missing replicas. Following the restoration of the zone, a subset of instances remained stuck in a REPAIRING state. For these affected instances, the data plane remained fully available and online for customer use, but control plane operations such as scaling and RDB exports were temporarily blocked. Google engineers manually mitigated this issue and fully repaired all affected instances within 24 hours. Next steps: The team is working on mechanisms to detect cases when the control plane and data plane get out of sync, then reconcile them so that the system can automatically recover. Apigee X Impact: During the power disruption, the Apigee X runtime in the region returned elevated HTTP 5xx errors for 7 hours and 19 minutes. Apigee X is designed to withstand a single-zone loss as its datastore keeps replicas across the region's three zones, but a scheduling misconfiguration had placed two of three replicas in the zone that lost power, so the datastore lost quorum and the runtime failed requests instead of failing over. The errors were concentrated in a small number of affected clusters rather than spread across the region. As remediation, engineers restored the affected replicas into nearby zones. Apigee Hybrid clusters were not severely impacted. Next steps: The engineering team has already audited the Apigee fleet to rebalance unevenly spread instances without downtime, and added monitoring to alert if datastore replicas drift across zones. Moving forward, the team will modify Apigee X to strictly enforce that every instance's three datastore replicas always run in three different zones. Additionally, the team is improving alert handling practices to help engineers better focus on affected services during similar outages. Apigee Edge Impact: During the power disruption, Apigee Edge clusters that had a single node in the affected zone successfully auto-recovered once power was restored. However, clusters with multiple nodes in the affected zone experienced extended recovery times. The automated node replacement system didn't handle multiple nodes going down simultaneously. This required manual intervention by Google engineers to restore the affected nodes. Furthermore, during this manual recovery, some virtual machines experienced a timeout when attempting to reach external OS repositories, causing them to boot with an older binary version and further delaying full restoration. Next steps: The engineering team will implement retries and scope repository updates to prevent pre-hook timeouts when reaching external OS repositories during the boot process. The team is also investigating further multiple-node failure scenario resilience improvements. Cloud Bigtable Impact: Cloud Bigtable behaved as expected during the europe-west2-b power outage. All single-cluster instances located in europe-west2-b experienced expected zonal unavailability while the zone lost power and recovered. Multi-cluster instances configured with High Availability (HA) routing automatically failed over to healthy replica clusters in other zones where replica clusters and the control plane remained available with sufficient capacity to absorb the failover load. List of impacted services We did not identify any impact of the outage outside of the europe-west2 region. The following Google Cloud Platform services experienced impact within europe-west2: * AlloyDB for PostgreSQL * Apigee * BigQuery Data Transfer Service * Cloud Data Fusion * Cloud Filestore * Cloud Firestore * Cloud Load Balancing * Cloud Memorystore * Cloud NAT * Cloud Run * Cloud Spanner * Cloud Workstations * Datastream * Dialogflow CX * Google App Engine * Google BigQuery * Google Cloud Bigtable * Google Cloud Dataflow * Google Cloud Dataproc * Google Cloud NetApp Volumes * Google Cloud Pub/Sub * Google Cloud SQL * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Google SecOps (Security Information and Event Management [SIEM] and Security Orchestration, Automation, and Response [SOAR]) * Looker (Google Cloud core) * Looker Studio * Managed Airflow (Google Cloud Composer) * Managed Service for Apache Kafka * Memorystore for Memcached * Memorystore for Redis * Memorystore for Redis Cluster * Memorystore for Valkey * Network Connectivity Center * Persistent Disk * Private Service Connect * Speech-to-Text * Virtual Private Cloud (VPC) * VMware Engine

Read the full incident report →

Started Oct 19, 2026, 06:56 PM UTC· resolved Oct 19, 2026, 08:40 PM UTC

1. resolved Oct 20, 2026, 04:40 AM UTC

   Elevated error rates and latency for Directions, Distance Matrix and Routes APIs. The issue has been resolved for all affected projects as of Monday, 2026-10-19 13:40 PDT.

Read the full incident report →

Started Oct 17, 2026, 12:00 AM UTC· resolved Oct 17, 2026, 08:00 AM UTC

1. monitoring Oct 16, 2026, 11:23 PM UTC

   Network maintenance from 5PM to 8PM PT resulting in reduced capacity for all Android & iOS physical, and Arm emulator devices.

2. monitoring Oct 17, 2026, 03:32 AM UTC

   Network maintenance window extended by 2 hours, ETA 10PM. Reduced capacity for all Android & iOS physical, and Arm emulator devices.

Read the full incident report →

Started Oct 15, 2026, 09:00 PM UTC· resolved Oct 16, 2026, 02:20 AM UTC

1. monitoring Oct 15, 2026, 08:54 PM UTC

   Network maintenance from 2PM to 8PM PT resulting in reduced capacity for all Android & iOS physical, and Arm emulator devices.

Read the full incident report →

Started Oct 13, 2026, 02:13 PM UTC· resolved Oct 13, 2026, 04:30 PM UTC

1. monitoring Oct 13, 2026, 02:16 PM UTC

   Currently, no significant impact to customer queue latency observed.

2. resolved Oct 13, 2026, 04:47 PM UTC

   Capacity has been restored.

Read the full incident report →

Started Oct 08, 2026, 11:00 AM UTC· resolved Oct 09, 2026, 04:46 PM UTC

1. investigating Oct 08, 2026, 06:24 PM UTC

   Description: We are experiencing an issue with Google Cloud Storage beginning on Thursday, 2026-10-08 05:19 PDT. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2026-10-08 11:50 PDT with current details. Customer symptoms: * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors Workaround: None at this time.

2. investigating Oct 08, 2026, 06:34 PM UTC

   Description: We are experiencing an issue with Google Cloud Storage beginning on Thursday, 2026-10-08 05:19 PDT. Initial mitigations applied by our engineering team leading to observable improvements in error rates in our internal monitoring. Work remains underway to roll out further mitigations. We will provide an update by Thursday, 2026-10-08 12:00 PDT with current details. Customer symptoms: * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors Workaround: None at this time.

3. investigating Oct 08, 2026, 06:55 PM UTC

   Description: We are experiencing an issue with Google Cloud Storage beginning on Thursday, 2026-10-08 05:19 PDT. Initial mitigations applied by our engineering team leading to observable improvements in error rates in our internal monitoring beginning 10:57 PDT. Work remains underway to roll out further mitigations. We will provide an update by Thursday, 2026-10-08 12:45 PDT with current details. Customer symptoms: * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors Workaround: None at this time.

4. investigating Oct 08, 2026, 07:38 PM UTC

   Description: We are continuing to mitigate the issue with Google Cloud Storage in the us-central1 region that began on Thursday, 2026-10-08 05:19 PDT. The initial mitigations applied by our engineering team continue to show observable improvements in error rates. Our engineering team is actively monitoring the environment and rolling out further mitigations to fully stabilize the service. We will provide an update by Thursday, 2026-10-08 14:00 PDT with current details. Customer symptoms: * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors Workaround: None at this time.

5. investigating Oct 08, 2026, 08:48 PM UTC

   Description: We are continuing to mitigate the issue with Google Cloud Storage in the us-central1 region that began on Thursday, 2026-10-08 05:19 PDT. The initial mitigations applied by our engineering team continue to show observable improvements in error rates. Our engineering team is actively monitoring the environment and rolling out further mitigations to fully stabilize the service. We will provide an update by Thursday, 2026-10-08 16:00 PDT with current details. Customer symptoms: * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors Workaround: None at this time.

6. resolved Oct 09, 2026, 04:44 PM UTC

   Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https\://cloud.google.com/support (https://cloud.google.com/support) Date/Time of the Issue (All time US/Pacific) Incident Start: 08 October 2026 04:00 Incident End: 08 October 2026 16:30 Duration: 12 hours, 30 minutes Summary On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates in the us-central1 region for a duration of 12 hours, 30 minutes. We are taking immediate steps to ensure this doesn’t happen again. Preliminary Root Cause Google Cloud Storage relies on a highly distributed backend architecture to process and store incoming data. During the incident window, specific storage backend nodes within the us-central1 region experienced a localized capacity shortfall. The sequence of events leading to customer impact was as follows: * A traffic sorting optimization intended to address localized capacity constraints caused severe resource exhaustion on specific backend storage nodes in the affected region. * This resource exhaustion caused backend services handling object data ingestion to begin timing out. * As a result, incoming upload and write requests routed to these nodes experienced elevated latency and failed to complete, resulting in errors surfaced to customers. * Concurrently, a separate issue with the underlying database service contributed to additional metadata processing errors during a portion of the incident window. Google engineers have begun a full root cause analysis (RCA) and will provide additional information once it is available. Remediation The issue was initially detected by Google engineering teams actively monitoring system telemetry early in the incident window. To mitigate the issue, Google engineers executed the following step-by-step strategy: * Engineers initially attempted to dynamically allocate additional emergency capacity to the affected region, but this was blocked by systemic regional capacity limits. * To relieve immediate pressure, engineers selectively drained traffic from the most overloaded backend storage nodes. * A configuration change was deployed to disable the traffic sorting optimization, significantly reducing resource overhead on the storage backend. * For the concurrent metadata issue, engineers rolled back a recent update to the underlying metadata database service. Once traffic levels stabilized and resource utilization returned to normal baseline levels, the previously drained nodes were restored to active service, successfully restoring normal operations. Description of Impact On 08 October, 2026, from 04:00 to 16:30 US/Pacific, customers utilizing the us-central1 region experienced elevated latency and error rates during Cloud Storage upload and write operations. * Customers primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts. * A minor impact was also observed on read operations during this timeframe. * Cloud Storage operations in all other regions, as well as all other Google Cloud services, remained unaffected by this specific event.

7. resolved Oct 14, 2026, 07:15 AM UTC

   Incident Report Date/Time of the Issue (All time US/Pacific) Incident first occurrence: Incident Start: 08 October 2026 04:00 Incident End: 08 October 2026 16:30 Duration: 12 hours, 30 minutes Incident recurrence: Recurrence Start: 09 October 2026 04:05 Recurrence End: 09 October 2026 09:46 Duration: 5 hours, 41 minutes Total Duration: 18 hours, 11 minutes Summary On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates while performing upload or write operations in the us-central1 region for a duration of 12 hours, 30 minutes. This was followed by a recurrence on Friday, 09 October 2026 in the us-central1-b zone for a duration of 5 hours, 41 minutes. We are taking immediate steps to ensure this doesn’t happen again. Root Cause Google Cloud Storage relies on a highly distributed backend architecture to process and store incoming data. During the incident window, specific storage backend nodes within the us-central1 region experienced a localized capacity shortfall due to increased traffic. The sequence of events leading to customer impact was as follows: * A traffic sorting optimization configuration update to the Google Cloud Storage load balancing layer intended to address localized capacity constraints caused severe resource exhaustion on specific storage backend nodes in the affected region by inadvertently bypassing the standard capacity-aware routing logic, resulting in a severe load imbalance. * Consequently, a small fraction of storage backend nodes in the us-central1 region received an overwhelming proportion of incoming write traffic despite having low available serving capacity. This concentration of traffic led to resource exhaustion on the affected backend servers causing backend services handling object data ingestion to begin timing out. * Because the underlying instances could not process the high volume of requests quickly enough, the incoming upload and write requests routed to these nodes experienced elevated latency and failed to complete, resulting in errors surfaced to customers. * Concurrently, a separate issue with the underlying database service contributed to additional metadata processing errors during a portion of the incident window. * The recurrence of the issue on October 9 was caused by an incomplete rollback of an upload planner feature, which broke the load balancing mechanism and overloaded a few smaller storage backend nodes. Remediation and Prevention The issue was initially detected on 08 October 2026 at 04:50 by Google engineering teams actively monitoring system telemetry early in the incident window. To mitigate the issue, Google engineers executed the following step-by-step strategy: * Engineers initially attempted to dynamically allocate additional emergency capacity to the affected region, but this was blocked by systemic regional capacity limits. * To relieve immediate pressure, engineers selectively drained traffic from the most overloaded storage backend nodes. * A configuration change was deployed to disable the traffic sorting optimization, significantly reducing resource overhead on the storage backend. * For the concurrent metadata issue, engineers rolled back a recent update to the underlying metadata database service. * While these steps provided temporary relief, a recurrence of the load imbalance required further traffic redirection and draining of affected storage components. The incident was permanently resolved when engineering fully reverted the problematic load balancing configuration change, restoring the previous capacity-aware routing behavior and allowing traffic to distribute evenly across the region. Once traffic levels stabilized and resource utilization returned to normal baseline levels, the previously drained nodes were restored to active service, successfully restoring normal operations. Preventative actions: To ensure this issue does not recur, we are implementing the following improvements: - Safe Configuration Rollouts: We are updating our deployment protocols to ensure that load balancing logic changes are rigorously validated in alternative, smaller regions before being rolled out to larger, highly utilized regions. - Intelligent Operational Controls: We are enhancing the storage load balancing architecture to strictly enforce capacity constraints, ensuring that traffic distribution logic cannot bypass pre-spilling safeguards and route traffic to clusters lacking sufficient serving resources. - Resilient Scaling Logic: We are implementing targeted improvements to accelerate internal data movement within highly utilized storage cells, enabling the system to process data more efficiently and handle temporary load spikes with greater resilience. Description of Impact On 08 October 2026, from 04:00 to 16:30 in us-central1 region and 09 October 2026, from 04:05 to 09:46 in us-central1-b zone, customers experienced elevated latency and error rates during Cloud Storage upload and write operations. * Upload & Write Failures: Customers primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts. * Increased Write Latency: Write operations that did succeed experienced significantly higher latencies than normal. * Cloud Storage operations in all other regions, as well as all other Google Cloud services, remained unaffected by this specific event.

Read the full incident report →

Started Oct 08, 2026, 10:31 AM UTC· resolved Oct 08, 2026, 08:00 PM UTC

1. investigating Oct 08, 2026, 01:38 PM UTC

   We're investigating reports of an issue with AdMob. We will provide more information shortly. The affected users are able to access AdMob, but are seeing error messages, high latency, and/or other unexpected behavior. Affected AdMob Interstitial video ads fail to render, causing a solid black screen and an unresponsive close button on both iOS and Android. Consequently, users are unable to dismiss the ad. There is no workaround other than force closing the affected application to exit the black screen.

2. resolved Oct 08, 2026, 08:08 PM UTC

   The problem with AdMob has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. The affected users are able to access AdMob, but are seeing error messages, high latency, and/or other unexpected behavior. Affected AdMob Interstitial video ads fail to render, causing a solid black screen and an unresponsive close button on both iOS and Android. Consequently, users are unable to dismiss the ad.

Read the full incident report →

Started Oct 07, 2026, 04:30 PM UTC· resolved Oct 09, 2026, 02:06 AM UTC

1. investigating Oct 08, 2026, 04:36 PM UTC

   We are investigating an issue with Firebase Hosting where projects—including those on the Blaze plan—may intermittently serve HTTP 509 "Bandwidth Quota Exceeded" errors or fail deployments with HTTP 429 quota errors. Workaround / Immediate Mitigation: Affected customers can mitigate the issue immediately by ensuring that the Firebase Hosting API is enabled for their project in Google Cloud Console: https://console.cloud.google.com/apis/library/firebasehosting.googleapis.com?project=_ Our engineering team is actively investigating the issue and working on a resolution.

2. investigating Oct 08, 2026, 08:21 PM UTC

   A rollback has completed that should mitigate the bandwidth quota errors. Deployments that are failing due to quota may still be impacted as the rollback for that service continues. Ensuring the API is enabled is still a mitigation for more immediate resolution.

3. resolved Oct 09, 2026, 09:44 PM UTC

   The issue has been resolved.

Read the full incident report →

Started Oct 03, 2026, 06:00 PM UTC· resolved Oct 04, 2026, 12:57 AM UTC

1. monitoring Oct 03, 2026, 08:45 PM UTC

   We're investigating reports of an issue with Google Ad Manager. We will provide more information shortly. The affected users are able to access Google Ad Manager, but may not have access to the most recent data. Google Ad Manager Reporting tool is currently experiencing data delays. Data Transfer is unaffected.

2. monitoring Oct 04, 2026, 12:57 AM UTC

   The problem with Google Ad Manager has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. Google Ad Manager Reporting data is updating as expected again.

Read the full incident report →

Started Sep 29, 2026, 12:41 AM UTC· resolved Sep 29, 2026, 02:52 AM UTC

1. investigating Sep 30, 2026, 12:57 AM UTC

   We're investigating reports of an issue with Google Analytics. We will provide more information shortly.

2. investigating Sep 30, 2026, 12:59 AM UTC

   Google Analytics service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.

3. resolved Sep 30, 2026, 01:44 AM UTC

   The problem with Google Analytics has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Read the full incident report →

Started Sep 28, 2026, 04:52 PM UTC· resolved Sep 29, 2026, 03:23 PM UTC

1. monitoring Sep 28, 2026, 06:56 PM UTC

   Summary: [Maps SDK] Fatal Exception: m140.fqd: GL context is unusable again / glLinkProgram failed "GVI && cannot compute gv size for oob" on Qualcomm Adreno GPUs (maps_core_dynamite v263020506) Description: We are also posting this incident to the Google Maps Platform Status Dashboard (https://status.cloud.google.com/maps-platform/ ) while the fix rolls out. Based on our telemetry, this issue affects up to 0.15% of Maps SDK map loads in impacted apps, though the rate varies significantly depending on each app's device GPU distribution (Qualcomm Adreno 6xx/7xx). Diagnosis: A shader linking error crashes the host app as soon as a Google Map is loaded. Workaround: None at this time.

2. monitoring Sep 29, 2026, 08:21 AM UTC

   Summary: [Maps SDK] Fatal Exception: m140.fqd: GL context is unusable again / glLinkProgram failed "GVI && cannot compute gv size for oob" on Qualcomm Adreno GPUs (maps_core_dynamite v263020506) Description: Our engineering team has identified the root cause of the crash spike. It was triggered by an unexpected interaction between a recent Android WebView configuration update and the Maps SDK on devices with Qualcomm Adreno 6xx/7xx GPUs. We have successfully initiated a server-side rollback of this WebView configuration. The safe configuration is currently deploying globally. Because Android WebView fetches configuration updates in the background, affected devices will recover automatically as they sync the updated configuration over the network. We are actively monitoring the rollout of this mitigation and will provide another update once it has further progressed. Diagnosis: A shader linking error crashes the host app as soon as a Google Map is loaded. Workaround: For end users who are currently experiencing persistent crashes and require immediate relief, clearing the host app's storage will force a fresh fetch of the safe configuration and break the crash loop. Please note that clearing the app cache alone is ineffective; it requires clearing the app storage (which will reset local app data).

3. monitoring Sep 29, 2026, 11:42 AM UTC

   Summary: [Maps SDK] Fatal Exception: m140.fqd: GL context is unusable again / glLinkProgram failed "GVI && cannot compute gv size for oob" on Qualcomm Adreno GPUs (maps_core_dynamite v263020506) Description: The server-side rollback of the Android WebView configuration has reached 100% global deployment. We are currently monitoring telemetry to verify the impact as affected devices begin to sync the safe configuration over the network. Because Android WebView fetches these updates in the background, client recovery is expected to occur gradually over the coming hours and days. We will keep this incident open and provide another update once we can confirm a reduction in crash volumes across the ecosystem. Diagnosis: A shader linking error crashes the host app as soon as a Google Map is loaded. Workaround: For end users who are currently experiencing persistent crashes and require immediate relief, clearing the host app's storage will force a fresh fetch of the safe configuration and break the crash loop. Please note that clearing the app cache alone is ineffective; it requires clearing the app storage (which will reset local app data).

4. resolved Sep 29, 2026, 03:23 PM UTC

   The issue with Maps SDK for Android, Navigation SDK has been resolved for all affected users as of Tuesday, 2026-09-29 00:00 PDT. However due to caching of feature flags we expect to continue to see sparse crash reports for a few days. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue.

Read the full incident report →

Started Sep 25, 2026, 03:47 PM UTC· resolved Sep 25, 2026, 04:40 PM UTC

1. monitoring Sep 25, 2026, 04:29 PM UTC

   Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2026-09-25 10:30 PDT. Symptoms: Customers may experience an endless loading screen when attempting to access Google Workspace Studio on the web. Workaround: None at this time.

2. resolved Sep 25, 2026, 04:52 PM UTC

   Description: The issue with Google Workspace Studio has been resolved for all affected users as of Friday, 2026-09-25 09:40 PDT. From preliminary analysis, the root cause of the issue is a recent rollout. Our engineering team has rolled back which mitigated the issue. We thank you for your patience while we worked on resolving the issue. Symptoms: Customers may have experienced an endless loading screen when attempting to access Google Workspace Studio on the web. Workaround: The issue is now mitigated.

Read the full incident report →

Started Sep 24, 2026, 04:15 PM UTC· resolved Oct 08, 2026, 08:00 AM UTC

1. monitoring Sep 24, 2026, 04:15 PM UTC

   Released the September 2026 spam update , which applies globally and to all languages. The rollout may take up to two weeks to complete.

2. resolved Oct 08, 2026, 08:37 AM UTC

   The rollout was complete as of October 8, 2026.

Read the full incident report →

Started Sep 23, 2026, 08:40 PM UTC· resolved Sep 23, 2026, 09:37 PM UTC

1. monitoring Sep 23, 2026, 08:45 PM UTC

   We're investigating reports of an issue with Google Ad Manager. We will provide more information shortly. The affected users are able to access Google Ad Manager, but are seeing error messages, high latency, and/or other unexpected behavior. Google Ad Manager users are not able to create master/companion line items

2. resolved Sep 23, 2026, 09:37 PM UTC

   The problem with Google Ad Manager has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. The affected users are able to access Google Ad Manager, but are seeing error messages, high latency, and/or other unexpected behavior. Google Ad Manager users are now able to create master/companion line items

Read the full incident report →

Started Sep 22, 2026, 10:00 AM UTC· resolved Sep 22, 2026, 11:30 PM UTC

1. investigating Sep 22, 2026, 04:51 PM UTC

   Firebase Test Lab is experiencing availability issues with ARM emulator devices.

2. investigating Sep 22, 2026, 09:28 PM UTC

   ARM emulator availability issues have been mitigated and full traffic capacity has been restored.

Read the full incident report →

Started Sep 22, 2026, 02:34 AM UTC· resolved Sep 22, 2026, 04:40 AM UTC

1. investigating Sep 22, 2026, 04:06 AM UTC

   Firebase App Hosting is having issues actuating Run. Services that are already running should be unaffected, but new builds or rollouts may be having issues.

2. resolved Sep 22, 2026, 05:17 AM UTC

   Firebase App Hosting's actuations of Cloud Run have been restored and builds should be working as expected now.

Read the full incident report →

Started Sep 21, 2026, 09:38 PM UTC· resolved Sep 22, 2026, 03:05 PM UTC

1. investigating Sep 22, 2026, 04:25 PM UTC

   Firebase Hosting has received report of sites being unavailable in some regions.

2. resolved Sep 22, 2026, 04:26 PM UTC

   The connectivity issues seem to be resolved and we are seeing requests complete from those regions once again. We will continue to monitor but believe the situation has recovered.

Read the full incident report →

Started Sep 21, 2026, 07:39 PM UTC· resolved Sep 21, 2026, 11:39 PM UTC

1. investigating Sep 21, 2026, 07:40 PM UTC

   We are experiencing high test latencies for ARM tests due to reduced capacity and high demand.

2. investigating Sep 21, 2026, 07:46 PM UTC

   We are rolling out a new server update to mitigate the issue.

Read the full incident report →

Started Sep 18, 2026, 05:00 AM UTC· resolved Sep 24, 2026, 06:00 PM UTC

1. monitoring Sep 22, 2026, 09:38 PM UTC

   Description: Our engineering team has identified the root cause and is actively working on mitigating the issue. We will provide more information by Tuesday, 2026-09-22 18:00 PDT. Symptoms: - Users on the Gmail Android app are finding that their accounts unexpectedly appear logged out. - Upon re-login, the process completes but the app fails to recognize the account. - Users receive an error message stating the account is already added. Workaround: The workaround for the Gmail app issue is to uninstall recent application updates and revert to the system's base version of Gmail. Optionally reinstall the latest version of the Gmail app from play store. Alternatively, users can access https://mail.google.com on their mobile browsers.

2. monitoring Sep 23, 2026, 12:49 AM UTC

   Description: Our engineering team has identified the root cause and is actively working on mitigating the issue. We will provide more information by Tuesday, 2026-09-22 20:30 PDT. Symptoms: - Users on the Gmail Android app are finding that their accounts unexpectedly appear logged out. - Upon re-login, the process completes but the app fails to recognize the account. - Users receive an error message stating the account is already added. Workaround: The workaround for the Gmail app issue is to uninstall recent application updates and revert to the system's base version of Gmail. Optionally reinstall the latest version of the Gmail app from play store. Alternatively, users can access https://mail.google.com on their mobile browsers.

3. monitoring Sep 23, 2026, 02:56 AM UTC

   Description: Our engineering team has identified the root cause and is actively working on mitigating the issue. We will provide more information by Wednesday, 2026-09-23 10:00 PDT. Symptoms: - Users on the Gmail Android app may observe sudden, unexpected account logout. - Upon re-login, the process completes but the app may fail to recognize the account. - Users may receive an error message stating the account is already added. Workaround: The workaround for the Gmail app issue is to uninstall recent application updates and revert to the system's base version of Gmail. Optionally reinstall the latest version of the Gmail app from play store. Alternatively, users can access https://mail.google.com on their mobile browsers.

4. monitoring Sep 23, 2026, 04:35 PM UTC

   Description: Our engineering team has identified the root cause and is actively working on mitigating the issue. We will provide more information by Wednesday, 2026-09-23 17:00 PDT. Symptoms: - Users on the Gmail Android app may observe sudden, unexpected account logout. - Upon re-login, the process completes but the app may fail to recognize the account. - Users may receive an error message stating the account is already added. Workaround: The workaround for the Gmail app issue is to uninstall recent application updates and revert to the system's base version of Gmail. Optionally reinstall the latest version of the Gmail app from play store. Alternatively, users can access https://mail.google.com on their mobile browsers.

5. monitoring Sep 24, 2026, 12:10 AM UTC

   Description: Our engineering team has identified the root cause and is rolling out a fix. The updated version of the Gmail app is currently available to an initial group of users to ensure stability, before a global rollout is initiated. We will provide our next update by Thursday, 2026-09-24 10:00 PDT Symptoms: - Users on the Gmail Android app may observe sudden, unexpected account logout. - Upon re-login, the process completes but the app may fail to recognize the account. - Users may receive an error message stating the account is already added. Workaround: The workaround for the Gmail app issue is to uninstall recent application updates and revert to the system's base version of Gmail. Optionally reinstall the latest version of the Gmail app from play store. Alternatively, users can access https://mail.google.com on their mobile browsers.

6. monitoring Sep 24, 2026, 04:02 PM UTC

   Description: Our engineering team has identified the root cause and is rolling out a fix. The updated version of the Gmail app is currently available to an initial group of users to ensure stability, before a global rollout is initiated. We will provide our next update by Thursday, 2026-09-24 16:00 PDT. Symptoms: - Users on the Gmail Android app may observe sudden, unexpected account logout. - Upon re-login, the process completes but the app may fail to recognize the account. - Users may receive an error message stating the account is already added. Workaround: The workaround for the Gmail app issue is to uninstall recent application updates and revert to the system's base version of Gmail. Optionally reinstall the latest version of the Gmail app from play store. Alternatively, users can access https://mail.google.com on their mobile browsers.

7. monitoring Sep 24, 2026, 10:29 PM UTC

   Description: Our engineering team has identified the root cause and is rolling out a fix. The updated version of the Gmail app is currently available to an initial group of users to ensure stability, before a global rollout is initiated. We will provide our next update by Thursday, 2026-09-24 18:30 PDT. Symptoms: - Users on the Gmail Android app may observe sudden, unexpected account logout. - Upon re-login, the process completes but the app may fail to recognize the account. - Users may receive an error message stating the account is already added. Workaround: The workaround for the Gmail app issue is to uninstall recent application updates and revert to the system's base version of Gmail. Optionally reinstall the latest version of the Gmail app from play store. Alternatively, users can access https://mail.google.com on their mobile browsers.

8. resolved Sep 25, 2026, 12:04 AM UTC

   Description: The issue with Gmail Android App has been resolved for all affected users as of Thursday, 2026-09-24 11:00 PDT. We thank you for your patience while we worked on resolving the issue. Symptoms: Users on the Gmail Android app may observe sudden, unexpected account logout. Upon re-login, the process completes but the app may fail to recognize the account. Users may receive an error message stating the account is already added. Workaround: A fix has been rolled out with the latest version of the Gmail Android app. Users can update to the latest version within the Play Store to resolve this issue.

Read the full incident report →

Started Sep 18, 2026, 12:45 AM UTC· resolved Sep 18, 2026, 02:10 AM UTC

1. investigating Sep 18, 2026, 12:50 AM UTC

   We're investigating reports of an issue with Google Ad Manager. We will provide more information shortly. The affected users are able to access Google Ad Manager, but are seeing error messages, high latency, and/or other unexpected behavior. Google Ad Manager Reporting tool is currently experiencing data delays (including API). Data Transfer is unaffected.

2. resolved Sep 18, 2026, 02:17 AM UTC

   The problem with Google Ad Manager has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. The affected users are able to access Google Ad Manager, but are seeing error messages, high latency, and/or other unexpected behavior. Google Ad Manager Reporting data is refreshing as expected again.

Read the full incident report →

Started Sep 16, 2026, 06:00 PM UTC· resolved Sep 16, 2026, 08:30 PM UTC

1. investigating Sep 16, 2026, 07:25 PM UTC

   Starting at 2PM ET, Crashlytics and Performance Monitoring console started experiencing low availability. We are working to remediate ASAP.

2. resolved Sep 16, 2026, 08:36 PM UTC

   Disruption is resolved at 13:30 PDT

Read the full incident report →