Google Outage History

Started Sep 01, 2026, 02:42 PM UTC· resolved Sep 01, 2026, 06:52 PM UTC

1. investigating Sep 01, 2026, 04:03 PM UTC

   Description: We are experiencing a networking issue starting Tuesday, 2026-09-01 07:41 US/Pacific. The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Our network engineering team has put in mitigations and we are seeing services starting to recover. We will provide more information by Tuesday, 2026-09-01 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Customer symptoms: Customers are experiencing elevated packet loss and errors. * BigQuery * Cloud Dataflow * Cloud SQL * App Engine * Cloud Run * AlloyDB * Cloud Bigtable * Cloud Spanner * Compute Engine * Cloud VPN * Cloud Interconnect * Cloud NAT * Cloud Filestore * Cloud Hybrid Connectivity * Virtual Private Cloud * Google Kubernetes Engine * Looker (Google Cloud core) Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones.

2. investigating Sep 01, 2026, 04:19 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Following network-level mitigation efforts, the underlying network infrastructure has fully recovered. Product engineering teams across Cloud are currently validating the recovery status of individual services. We will provide more information by Tuesday, 2026-09-01 10:00 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * BigQuery * Cloud Dataflow * Cloud SQL * App Engine * Cloud Run * AlloyDB * Cloud Bigtable * Cloud Spanner * Compute Engine * Cloud VPN * Cloud Interconnect * Cloud NAT * Cloud Filestore * Cloud Hybrid Connectivity * Virtual Private Cloud * Google Kubernetes Engine * Looker (Google Cloud core) * Apigee * Google SecOps SOAR Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones.

3. investigating Sep 01, 2026, 05:09 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. We will provide more information by Tuesday, 2026-09-01 11:00 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones.

4. investigating Sep 01, 2026, 06:01 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. Routine network maintenance triggered issues in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected.

5. investigating Sep 01, 2026, 06:36 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered. The impacted products Cloud Run and App Engine are also seeing recovery and the errors have lowered pre-incident levels. Engineering continues to monitor and validate. Routine network fabric path maintenance triggered unexpected issues in two clusters in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected.

6. resolved Sep 01, 2026, 07:12 PM UTC

   Description: The issue was affecting a portion of the us-central1-b zone. The remaining zones in the region were unaffected. All the impacted products have recovered. From preliminary analysis, routine network fabric path maintenance triggered unexpected issues in one cluster in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We thank you for your patience while we worked on resolving the issue. Customer symptoms: Customers experienced elevated packet loss and errors. Workaround: The issue has been mitigated.

7. resolved Sep 03, 2026, 05:47 PM UTC

   Affected services and features: * Google Compute Engine: Inability to access virtual machines externally, and inability for VMs to reach remote resources. * Google Kubernetes Engine: Unreachable regional/zonal clusters and nodes within the affected zone, prompting delayed cluster recovery times. * Cloud Run / Google App Engine: Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific. * Cloud SQL, AlloyDB for PostgreSQL, Cloud Spanner, Google Cloud Bigtable, Cloud Filestore: Data access and connectivity severed for instances localized strictly to the impacted infrastructure. * Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect: Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone.

8. resolved Sep 10, 2026, 09:20 PM UTC

   Affected services and features: * Google Compute Engine: Inability for users in the affected portion of us-central1-b and us-central1-f zones to access virtual machines externally, and inability for VMs to reach remote resources. * Cloud SQL, AlloyDB for PostgreSQL, Google Cloud Bigtable, Cloud Filestore: Data access and connectivity severed for instances localized strictly to the impacted infrastructure. * Cloud Spanner, Cloud Firestore: Elevated Remote Procedure Call (RPC) error rates, write timeouts, and latency spikes for the nam5 multi-region instances due to severed connectivity to components running in the impacted zone. * Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect: Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone. * Google Kubernetes Engine (GKE): Between 07:40 and 09:15 US/Pacific on 2026-09-01, customers experienced errors and timeouts on requests to their GKE cluster control planes (Kubernetes API servers) in us-central1. This could have caused new or updated workloads to fail scheduling and Kubernetes API operations to fail. * Cloud Run / Google App Engine: Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific. Customer impact: Customers with resources hosted in the specific affected clusters within us-central1-b and us-central1-f experienced a complete loss of network connectivity starting at 07:41 US/Pacific. During this time, virtual machines and associated services became unreachable from the internet and from other Google Cloud regions, and those internal resources could not initiate outbound connections. The issue was detected almost immediately and engineers took traffic diversion actions at 07:45 and 08:00 US/Pacific to reduce impact to regional products. By 08:50 US/Pacific, the majority of physical connections were restored and traffic began recovering, and the traffic diversion actions were removed at 09:19 US/Pacific. Full recovery across nearly all affected services and long-running operations was confirmed by 11:52 US/Pacific. Because the network isolation was strictly limited to specific clusters within us-central1-b and us-central1-f, multi-zonal deployments correctly utilizing redundancy across other zones in us-central1 were largely able to bypass the physical hardware failure and continue serving traffic. Customers with projects in both us-central1-b and us-central1-f would not have seen multi-zonal impact - at most one of their zones would have capacity in the affected data center.

9. resolved Sep 29, 2026, 04:09 PM UTC

   Addendum to Incident Report This addendum extends the Detailed Description of Impact from the previous message. Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected. Between 07:41 and 11:52 US/Pacific on Tuesday, September 1, 2026, the following products were affected: - Compute Engine VMs: VMs running in the affected data center lost network connectivity to resources located outside of their datacenter. From a Compute Engine perspective, this incident only affected a subset of VMs in us-central1-b and us-central1-f. - Zonal and Regional Products Based on Compute Engine Products with dependencies on VMs in the affected datacenter experienced symptoms like connection timeouts or service unavailability because the underlying VMs lost network connectivity. Examples of products that are based on Compute Engine VMs include: Apigee, Cloud SQL, AlloyDB for PostgreSQL, Dataflow, Cloud Filestore, and Looker. - Google Kubernetes Engine (GKE): GKE clusters whose control plane had dependencies on Compute Engine VMs in the affected clusters experienced errors and timeouts when administrators or nodes attempted to connect to the Kubernetes API server. Consequently, deployments might have been delayed, Kubernetes API operations might have failed, and new nodes might not have registered with the control plane (causing node pool operational delays). - Cloud Run and App Engine: Workloads in the us-central1 region experienced elevated serving latency, 5xx aborted request errors, and outbound connectivity degradation. To prevent severe request failures, traffic was diverted to other clusters. Diverting requests to other clusters temporarily caused elevated latency or errors in those clusters.Once the impacted cluster was restored, workload demand for these new backends created a 'thundering herd' of concurrent instance initializations. This sudden surge bypassed warm caches and overwhelmed internal file-serving tiers, extending symptoms until emergency capacity was provisioned. - Other Google APIs and Services, in us-central1 or a multi-region that includes us-central1, were affected if they had dependencies on software tasks in the affected clusters and weren't able to use software tasks in an unaffected cluster. Symptoms included connection timeouts or service unavailability. Examples of other Google APIs and services are: Google Cloud Bigtable, BigQuery, Google SecOps SOAR, Cloud Spanner, and Cloud Firestore. For some services, symptoms were limited to temporary latency spikes and service unavailability until workloads shifted to unaffected clusters. - Private Service Connect: Clients (in any region) experienced connectivity issues involving PSC endpoints with dependencies in us-central1, in specific circumstances: - PSC endpoints for Google APIs were affected in the same way as noted in the Google APIs and services section. - Producer/consumer PSC endpoints were affected if the producer had load balancers or backend VMs in the affected clusters of us-central1. - Hybrid Connectivity: Some Cloud Router, Cloud VPN, and Cloud Interconnect VLAN attachments in us-central1 were affected in specific circumstances: - Cloud Router BGP software tasks for some Cloud Routers in us-central1 experienced a Cloud Router maintenance event if their BGP software tasks were running in the affected clusters. - Cloud VPN tunnel software tasks running in the affected clusters lost network connectivity. This caused some Cloud VPN tunnels in us-central1 to go down until a replacement tunnel task in a different cluster was assigned. - Resources (in any region) sending packets through Cloud Interconnect VLAN attachments in us-central1 might have experienced intermittent connectivity issues in certain situations where network flows weren’t already programmed. - Envoy-based load balancers and other products: Load balancers and other products that depended on managed Envoy tasks of a proxy-only subnet in us-central1 experienced increased latency or connectivity issues if enough Envoy tasks were in the affected clusters. The following products were affected: - Regional external application load balancers in us-central1 - Regional internal application load balancers in us-central1 - Cross-region internal application load balancers with proxies in us-central1 - Regional external proxy network load balancers in us-central1 - Regional internal proxy network load balancers in us-central1 - Cross-region internal proxy network load balancers with proxies in us-central1 - Secure Web Proxy in us-central1

Read the full incident report →

Started Sep 01, 2026, 02:42 PM UTC· resolved Sep 01, 2026, 06:52 PM UTC

1. investigating Sep 01, 2026, 04:03 PM UTC

   Description: We are experiencing a networking issue starting Tuesday, 2026-09-01 07:41 US/Pacific. The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Our network engineering team has put in mitigations and we are seeing services starting to recover. We will provide more information by Tuesday, 2026-09-01 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Customer symptoms: Customers are experiencing elevated packet loss and errors. * BigQuery * Cloud Dataflow * Cloud SQL * App Engine * Cloud Run * AlloyDB * Cloud Bigtable * Cloud Spanner * Compute Engine * Cloud VPN * Cloud Interconnect * Cloud NAT * Cloud Filestore * Cloud Hybrid Connectivity * Virtual Private Cloud * Google Kubernetes Engine * Looker (Google Cloud core) Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones.

2. investigating Sep 01, 2026, 04:19 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Following network-level mitigation efforts, the underlying network infrastructure has fully recovered. Product engineering teams across Cloud are currently validating the recovery status of individual services. We will provide more information by Tuesday, 2026-09-01 10:00 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * BigQuery * Cloud Dataflow * Cloud SQL * App Engine * Cloud Run * AlloyDB * Cloud Bigtable * Cloud Spanner * Compute Engine * Cloud VPN * Cloud Interconnect * Cloud NAT * Cloud Filestore * Cloud Hybrid Connectivity * Virtual Private Cloud * Google Kubernetes Engine * Looker (Google Cloud core) * Apigee * Google SecOps SOAR Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones.

3. investigating Sep 01, 2026, 05:09 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. We will provide more information by Tuesday, 2026-09-01 11:00 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones.

4. investigating Sep 01, 2026, 06:01 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. Routine network maintenance triggered issues in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected.

5. investigating Sep 01, 2026, 06:36 PM UTC

   Description: The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered. The impacted products Cloud Run and App Engine are also seeing recovery and the errors have lowered pre-incident levels. Engineering continues to monitor and validate. Routine network fabric path maintenance triggered unexpected issues in two clusters in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. Customer symptoms: Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore Workaround: * Customers with service layer retries may see successful completions as only a portion of the zone is affected.

6. resolved Sep 01, 2026, 07:12 PM UTC

   Description: The issue was affecting a portion of the us-central1-b zone. The remaining zones in the region were unaffected. All the impacted products have recovered. From preliminary analysis, routine network fabric path maintenance triggered unexpected issues in one cluster in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We thank you for your patience while we worked on resolving the issue. Customer symptoms: Customers experienced elevated packet loss and errors. Workaround: The issue has been mitigated.

7. resolved Sep 03, 2026, 05:47 PM UTC

   Affected services and features: * Google Compute Engine: Inability to access virtual machines externally, and inability for VMs to reach remote resources. * Google Kubernetes Engine: Unreachable regional/zonal clusters and nodes within the affected zone, prompting delayed cluster recovery times. * Cloud Run / Google App Engine: Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific. * Cloud SQL, AlloyDB for PostgreSQL, Cloud Spanner, Google Cloud Bigtable, Cloud Filestore: Data access and connectivity severed for instances localized strictly to the impacted infrastructure. * Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect: Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone.

8. resolved Sep 10, 2026, 09:20 PM UTC

   Affected services and features: * Google Compute Engine: Inability for users in the affected portion of us-central1-b and us-central1-f zones to access virtual machines externally, and inability for VMs to reach remote resources. * Cloud SQL, AlloyDB for PostgreSQL, Google Cloud Bigtable, Cloud Filestore: Data access and connectivity severed for instances localized strictly to the impacted infrastructure. * Cloud Spanner, Cloud Firestore: Elevated Remote Procedure Call (RPC) error rates, write timeouts, and latency spikes for the nam5 multi-region instances due to severed connectivity to components running in the impacted zone. * Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect: Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone. * Google Kubernetes Engine (GKE): Between 07:40 and 09:15 US/Pacific on 2026-09-01, customers experienced errors and timeouts on requests to their GKE cluster control planes (Kubernetes API servers) in us-central1. This could have caused new or updated workloads to fail scheduling and Kubernetes API operations to fail. * Cloud Run / Google App Engine: Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific. Customer impact: Customers with resources hosted in the specific affected clusters within us-central1-b and us-central1-f experienced a complete loss of network connectivity starting at 07:41 US/Pacific. During this time, virtual machines and associated services became unreachable from the internet and from other Google Cloud regions, and those internal resources could not initiate outbound connections. The issue was detected almost immediately and engineers took traffic diversion actions at 07:45 and 08:00 US/Pacific to reduce impact to regional products. By 08:50 US/Pacific, the majority of physical connections were restored and traffic began recovering, and the traffic diversion actions were removed at 09:19 US/Pacific. Full recovery across nearly all affected services and long-running operations was confirmed by 11:52 US/Pacific. Because the network isolation was strictly limited to specific clusters within us-central1-b and us-central1-f, multi-zonal deployments correctly utilizing redundancy across other zones in us-central1 were largely able to bypass the physical hardware failure and continue serving traffic. Customers with projects in both us-central1-b and us-central1-f would not have seen multi-zonal impact - at most one of their zones would have capacity in the affected data center.

9. resolved Sep 29, 2026, 04:09 PM UTC

   Addendum to Incident Report This addendum extends the Detailed Description of Impact from the previous message. Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected. Between 07:41 and 11:52 US/Pacific on Tuesday, September 1, 2026, the following products were affected: - Compute Engine VMs: VMs running in the affected data center lost network connectivity to resources located outside of their datacenter. From a Compute Engine perspective, this incident only affected a subset of VMs in us-central1-b and us-central1-f. - Zonal and Regional Products Based on Compute Engine Products with dependencies on VMs in the affected datacenter experienced symptoms like connection timeouts or service unavailability because the underlying VMs lost network connectivity. Examples of products that are based on Compute Engine VMs include: Apigee, Cloud SQL, AlloyDB for PostgreSQL, Dataflow, Cloud Filestore, and Looker. - Google Kubernetes Engine (GKE): GKE clusters whose control plane had dependencies on Compute Engine VMs in the affected clusters experienced errors and timeouts when administrators or nodes attempted to connect to the Kubernetes API server. Consequently, deployments might have been delayed, Kubernetes API operations might have failed, and new nodes might not have registered with the control plane (causing node pool operational delays). - Cloud Run and App Engine: Workloads in the us-central1 region experienced elevated serving latency, 5xx aborted request errors, and outbound connectivity degradation. To prevent severe request failures, traffic was diverted to other clusters. Diverting requests to other clusters temporarily caused elevated latency or errors in those clusters.Once the impacted cluster was restored, workload demand for these new backends created a 'thundering herd' of concurrent instance initializations. This sudden surge bypassed warm caches and overwhelmed internal file-serving tiers, extending symptoms until emergency capacity was provisioned. - Other Google APIs and Services, in us-central1 or a multi-region that includes us-central1, were affected if they had dependencies on software tasks in the affected clusters and weren't able to use software tasks in an unaffected cluster. Symptoms included connection timeouts or service unavailability. Examples of other Google APIs and services are: Google Cloud Bigtable, BigQuery, Google SecOps SOAR, Cloud Spanner, and Cloud Firestore. For some services, symptoms were limited to temporary latency spikes and service unavailability until workloads shifted to unaffected clusters. - Private Service Connect: Clients (in any region) experienced connectivity issues involving PSC endpoints with dependencies in us-central1, in specific circumstances: - PSC endpoints for Google APIs were affected in the same way as noted in the Google APIs and services section. - Producer/consumer PSC endpoints were affected if the producer had load balancers or backend VMs in the affected clusters of us-central1. - Hybrid Connectivity: Some Cloud Router, Cloud VPN, and Cloud Interconnect VLAN attachments in us-central1 were affected in specific circumstances: - Cloud Router BGP software tasks for some Cloud Routers in us-central1 experienced a Cloud Router maintenance event if their BGP software tasks were running in the affected clusters. - Cloud VPN tunnel software tasks running in the affected clusters lost network connectivity. This caused some Cloud VPN tunnels in us-central1 to go down until a replacement tunnel task in a different cluster was assigned. - Resources (in any region) sending packets through Cloud Interconnect VLAN attachments in us-central1 might have experienced intermittent connectivity issues in certain situations where network flows weren’t already programmed. - Envoy-based load balancers and other products: Load balancers and other products that depended on managed Envoy tasks of a proxy-only subnet in us-central1 experienced increased latency or connectivity issues if enough Envoy tasks were in the affected clusters. The following products were affected: - Regional external application load balancers in us-central1 - Regional internal application load balancers in us-central1 - Cross-region internal application load balancers with proxies in us-central1 - Regional external proxy network load balancers in us-central1 - Regional internal proxy network load balancers in us-central1 - Cross-region internal proxy network load balancers with proxies in us-central1 - Secure Web Proxy in us-central1

Read the full incident report →

Started Sep 01, 2026, 02:28 PM UTC· resolved Sep 01, 2026, 10:05 PM UTC

1. monitoring Sep 01, 2026, 04:28 PM UTC

   Google Ad Manager DAI service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change. The affected users are able to access Google Ad Manager DAI, but are seeing error messages, high latency, and/or other unexpected behavior. Ad Manager DAI streams return internal error. Engineering has identified the root cause and deployed a mitigation covering most streams, while work continues to address the remaining impact. There is currently no workaround.

2. resolved Sep 01, 2026, 10:05 PM UTC

   The problem with Google Ad Manager DAI has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. The affected users are able to access Google Ad Manager DAI, but are seeing error messages, high latency, and/or other unexpected behavior. Ad Manager DAI stream requests failing with internal errors (4xx or 5xx). The issue is now resolved.

Read the full incident report →

Started Sep 01, 2026, 08:54 AM UTC· resolved Sep 01, 2026, 11:43 AM UTC

1. investigating Sep 01, 2026, 11:11 AM UTC

   We're investigating reports of an issue with Play Billing Purchases and Play Subscriptions Renewal and Management. We will provide more information shortly. The affected users are able to access Play Billing Purchases and Play Subscriptions Renewal and Management, but are seeing error messages, high latency, and/or other unexpected behavior.

2. resolved Sep 01, 2026, 01:59 PM UTC

   The problem with Play Billing Purchases, Play Developer Subscriptions and In-App Purchases API, and Play Subscriptions Renewal and Management has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Read the full incident report →

Started Sep 01, 2026, 08:40 AM UTC· resolved Sep 01, 2026, 12:31 PM UTC

1. investigating Sep 01, 2026, 10:29 AM UTC

   We're aware of a problem with Firebase Cloud Messaging affecting its availability and latency in North America. We will provide an update by 04:30AM detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.

2. investigating Sep 01, 2026, 11:43 AM UTC

   Services are recovering. Next update at 05:30

3. resolved Sep 01, 2026, 12:53 PM UTC

   The problem has been resolved. All services in scope are back to normal.

Read the full incident report →

Started Sep 01, 2026, 08:00 AM UTC· resolved Sep 01, 2026, 12:22 PM UTC

1. investigating Sep 01, 2026, 03:44 PM UTC

   We're investigating reports of an issue with Play Integrity API. We will provide more information shortly.

2. resolved Sep 01, 2026, 03:50 PM UTC

   The problem with Play Integrity API has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Read the full incident report →

Started Aug 28, 2026, 05:50 PM UTC· resolved Aug 28, 2026, 08:34 PM UTC

1. monitoring Aug 28, 2026, 07:25 PM UTC

   Description: We are experiencing an intermittent issue with Google Chat Friday, 2026-08-28 10:25 PDT. Our engineering team has investigated the issue and are currently working on mitigating the issue. We will provide more information by Friday, 2026-08-28 14:15 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: Users are experiencing an issue where they receive notifications for new messages, but the actual messages fail to appear in their chat application unless the user refreshes their view Workaround: Users can refresh to see new messages

2. resolved Aug 28, 2026, 08:55 PM UTC

   Description: The issue with Google Chat has been resolved for all affected users as of Friday, 2026-08-28 13:45 PDT. From preliminary analysis the issue was triggered due to a recent change which was rolled back to mitigate the impact. We thank you for your patience while we worked on resolving the issue. Customer symptoms: Users were experiencing an issue where they received notifications for new messages, but the actual messages failed to appear in their chat application unless the user refreshed their view. Workaround: None as the issue is resolved.

3. resolved Sep 04, 2026, 03:43 AM UTC

   Google chat: * Delayed message rendering: New messages did not appear in active conversation streams automatically. Users were required to manually refresh their client or send a new message/react to force the stream to synchronize. * Stalled read states and inbox sorting: Background actions, such as updating unread badges and moving active spaces to the top of the inbox, experienced heavy internal error rates and were significantly delayed for users. * Search indexing delays: New messages sent during the outage window stalled in the indexing pipeline, experiencing delays in becoming searchable. Scope exclusions: * No Data Loss: At no point was customer data lost or compromised. All messages transmitted during the incident window were securely captured in primary storage and successfully delivered to conversation streams once the backlog cleared.

Read the full incident report →

Started Aug 20, 2026, 03:40 PM UTC· resolved Aug 20, 2026, 07:20 PM UTC

1. investigating Aug 20, 2026, 04:44 PM UTC

   Description: We are experiencing an issue with multiple products, beginning on Thursday, 2026-08-20 08:40 PDT. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2026-08-20 10:30 PDT with details. We apologize to all who are affected by the disruption. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: None at this time.

2. investigating Aug 20, 2026, 05:13 PM UTC

   Description: Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 10:45 PDT with details. Diagnosis / customer symptoms: Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 10:45 PDT with details. Workaround: We recommend customers to failover to other regions where feasible.

3. investigating Aug 20, 2026, 05:32 PM UTC

   Description: Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 11:00 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: We recommend customers to failover to other regions where feasible.

4. investigating Aug 20, 2026, 06:03 PM UTC

   Description: Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 11:45 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: No workarounds needed at this time.

5. investigating Aug 20, 2026, 06:53 PM UTC

   Description: Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 12:30 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: No workarounds needed at this time.

6. investigating Aug 20, 2026, 07:21 PM UTC

   Description: Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 13:30 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: No workarounds needed at this time.

7. resolved Aug 20, 2026, 07:37 PM UTC

   Description: We have mitigated the issue impacting multiple products in our us-west1 region as of Thursday, 2026-08-20 10:22 PDT. Our engineering teams have restored capacity on a planned optical maintenance that caused unexpected congestion in Dalles, Oregon metro /us-west1 region. Our systems stabilized and services have recovered once capacity was restored. Our teams are continuing to monitor for any residual impact. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue. Diagnosis / customer symptoms: Customers in us-west1 may have experienced timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: This issue is now mitigated.

8. resolved Aug 25, 2026, 06:20 AM UTC

   Affected services and features: The following services experienced elevated latencies and/or increased error rates, across their respective data planes and control planes: * AlloyDB * Apache Kafka * Apigee Edge Public Cloud * Apigee X * Artifact Registry * BigQuery & BigQuery Data Transfer Service * Cloud Build * Cloud Data Fusion * Cloud Dataflow * Cloud Filestore * Cloud Key Management Service (KMS) * Cloud Monitoring * Cloud Run * Cloud SQL * Contact Center AI Platform * Dataproc Metastore * Google App Engine * Google Cloud Bigtable * Google Cloud Pub/Sub * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Identity and Access Management (IAM) * Managed Airflow (Cloud Composer) *Managed Service for Apache Spark (Dataproc) Persistent Disk

9. resolved Aug 27, 2026, 09:45 PM UTC

   Affected services and features: The following services experienced elevated latencies and/or increased error rates, across their respective data planes and control planes: * AlloyDB * Apache Kafka * Apigee Edge Public Cloud * Apigee X * Artifact Registry * BigQuery & BigQuery Data Transfer Service * Cloud Build * Cloud Data Fusion * Cloud Dataflow * Cloud Filestore * Cloud Key Management Service (KMS) * Cloud Monitoring * Cloud Run * Cloud SQL * Contact Center AI Platform * Dataproc Metastore * Google App Engine * Google Cloud Bigtable * Google Cloud Pub/Sub * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Identity and Access Management (IAM) * Managed Airflow (Cloud Composer) * Managed Service for Apache Spark (Dataproc) * Persistent Disk The regional incident in us-west1 followed a structured three-phase recovery dictated by platform dependency layers: While core platform connectivity and live request serving were restored by 10:22 US/Pacific, certain services took additional time to fully recover due to asynchronous backlog processing and localized control-plane state reconciliation for a very small set of customers. For event-driven and pipeline services, inbound error rates dropped to 0 immediately, but some operations experienced elevated latency while workers processed through backlogs accumulated during the outage, clearing later for Cloud Pub/Sub, Cloud Build & Deploy, Cloud Dataflow, and Cloud Storage lifecycle deletions. Concurrently, while primary read/write traffic was healthy across the region, specific long-running lifecycle workflows required extra time to clear locks and reconcile distributed state machines. Compute Engine VM provisioning in zone us-west1-c and Cloud Filestore control plane instance allocation locks and resource validation checks took longer to normalize across regional storage backends.

Read the full incident report →

Started Aug 20, 2026, 03:40 PM UTC· resolved Aug 20, 2026, 07:20 PM UTC

1. investigating Aug 20, 2026, 04:44 PM UTC

   Description: We are experiencing an issue with multiple products, beginning on Thursday, 2026-08-20 08:40 PDT. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2026-08-20 10:30 PDT with details. We apologize to all who are affected by the disruption. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: None at this time.

2. investigating Aug 20, 2026, 05:13 PM UTC

   Description: Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 10:45 PDT with details. Diagnosis / customer symptoms: Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 10:45 PDT with details. Workaround: We recommend customers to failover to other regions where feasible.

3. investigating Aug 20, 2026, 05:32 PM UTC

   Description: Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 11:00 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: We recommend customers to failover to other regions where feasible.

4. investigating Aug 20, 2026, 06:03 PM UTC

   Description: Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 11:45 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: No workarounds needed at this time.

5. investigating Aug 20, 2026, 06:53 PM UTC

   Description: Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 12:30 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: No workarounds needed at this time.

6. investigating Aug 20, 2026, 07:21 PM UTC

   Description: Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 13:30 PDT with details. Diagnosis / customer symptoms: Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: No workarounds needed at this time.

7. resolved Aug 20, 2026, 07:37 PM UTC

   Description: We have mitigated the issue impacting multiple products in our us-west1 region as of Thursday, 2026-08-20 10:22 PDT. Our engineering teams have restored capacity on a planned optical maintenance that caused unexpected congestion in Dalles, Oregon metro /us-west1 region. Our systems stabilized and services have recovered once capacity was restored. Our teams are continuing to monitor for any residual impact. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue. Diagnosis / customer symptoms: Customers in us-west1 may have experienced timeouts, service degradations, errors, and elevated latencies across multiple products. Workaround: This issue is now mitigated.

8. resolved Aug 25, 2026, 06:20 AM UTC

   Affected services and features: The following services experienced elevated latencies and/or increased error rates, across their respective data planes and control planes: * AlloyDB * Apache Kafka * Apigee Edge Public Cloud * Apigee X * Artifact Registry * BigQuery & BigQuery Data Transfer Service * Cloud Build * Cloud Data Fusion * Cloud Dataflow * Cloud Filestore * Cloud Key Management Service (KMS) * Cloud Monitoring * Cloud Run * Cloud SQL * Contact Center AI Platform * Dataproc Metastore * Google App Engine * Google Cloud Bigtable * Google Cloud Pub/Sub * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Identity and Access Management (IAM) * Managed Airflow (Cloud Composer) *Managed Service for Apache Spark (Dataproc) Persistent Disk

9. resolved Aug 27, 2026, 09:45 PM UTC

   Affected services and features: The following services experienced elevated latencies and/or increased error rates, across their respective data planes and control planes: * AlloyDB * Apache Kafka * Apigee Edge Public Cloud * Apigee X * Artifact Registry * BigQuery & BigQuery Data Transfer Service * Cloud Build * Cloud Data Fusion * Cloud Dataflow * Cloud Filestore * Cloud Key Management Service (KMS) * Cloud Monitoring * Cloud Run * Cloud SQL * Contact Center AI Platform * Dataproc Metastore * Google App Engine * Google Cloud Bigtable * Google Cloud Pub/Sub * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Identity and Access Management (IAM) * Managed Airflow (Cloud Composer) * Managed Service for Apache Spark (Dataproc) * Persistent Disk The regional incident in us-west1 followed a structured three-phase recovery dictated by platform dependency layers: While core platform connectivity and live request serving were restored by 10:22 US/Pacific, certain services took additional time to fully recover due to asynchronous backlog processing and localized control-plane state reconciliation for a very small set of customers. For event-driven and pipeline services, inbound error rates dropped to 0 immediately, but some operations experienced elevated latency while workers processed through backlogs accumulated during the outage, clearing later for Cloud Pub/Sub, Cloud Build & Deploy, Cloud Dataflow, and Cloud Storage lifecycle deletions. Concurrently, while primary read/write traffic was healthy across the region, specific long-running lifecycle workflows required extra time to clear locks and reconcile distributed state machines. Compute Engine VM provisioning in zone us-west1-c and Cloud Filestore control plane instance allocation locks and resource validation checks took longer to normalize across regional storage backends.

Read the full incident report →

Started Aug 20, 2026, 02:30 PM UTC· resolved Aug 22, 2026, 01:50 PM UTC

1. monitoring Aug 20, 2026, 03:08 PM UTC

   Description: We are experiencing an issue with Firebase Performance Monitoring, beginning on Thursday, 2026-08-20 07:30 PDT. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2026-08-20 11:00 PDT with details. We apologize to all who are affected by the disruption. Diagnosis / customer symptoms: Network trace ingestion data from August 17, 2026 is delayed for Firebase Performance Monitoring customers. All other Performance Monitoring data and BigQuery exports remain accurate. There is no data loss associated with this event. Workaround: None at this time.

2. monitoring Aug 20, 2026, 06:39 PM UTC

   Description: We are experiencing an issue with Firebase Performance Monitoring, beginning on Thursday, 2026-08-20 07:30 PDT. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2026-08-20 15:30 PDT with details. Diagnosis / customer symptoms: Network trace ingestion data from August 17, 2026 is delayed for Firebase Performance Monitoring customers. All other Performance Monitoring data and BigQuery exports remain accurate. There is no data loss associated with this event. Workaround: None at this time.

3. monitoring Aug 20, 2026, 10:34 PM UTC

   Description: We are experiencing an issue with Firebase Performance Monitoring, beginning on Thursday, 2026-08-20 07:30 PDT. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2026-08-21 08:00 PDT with details. Diagnosis / customer symptoms: Network trace ingestion data from August 17, 2026 is delayed for Firebase Performance Monitoring customers. All other Performance Monitoring data and BigQuery exports remain accurate. There is no data loss associated with this event. Workaround: None at this time.

4. monitoring Aug 21, 2026, 12:21 PM UTC

   Description: We are experiencing an issue with Firebase Performance Monitoring, beginning on Thursday, 2026-08-20 07:30 PDT. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2026-08-21 20:00 PDT with details. Diagnosis / customer symptoms: Network trace ingestion data from August 18, 2026 is delayed for Firebase Performance Monitoring customers. All other Performance Monitoring data and BigQuery exports remain accurate. There is no data loss associated with this event. Workaround: None at this time.

5. monitoring Aug 21, 2026, 11:30 PM UTC

   Description: We are experiencing an issue with Firebase Performance Monitoring, beginning on Thursday, 2026-08-20 07:30 PDT. Our engineering team has identified the cause of this issue and is currently working on a mitigation. We will provide an update by Saturday, 2026-08-22 09:00 PDT with details. Diagnosis / customer symptoms: Network trace ingestion data for August 18, 2026 is delayed for Firebase Performance Monitoring customers. All other Performance Monitoring data and BigQuery exports remain accurate. There is no data loss associated with this event. Workaround: None at this time.

6. resolved Aug 22, 2026, 02:01 PM UTC

   Description: The issue with Firebase Performance Monitoring has been resolved for all affected users as of Saturday, 2026-08-22 06:50 PDT. We thank you for your patience while we worked on resolving the issue. Diagnosis / customer symptoms: The Firebase Dashboard has been fixed and currently is up to date, BigQuery is delayed for 2026-08-18 and 2026-08-19. All other Performance Monitoring data and BigQuery exports remain accurate. There is no data loss associated with this event. Workaround: None as the issue has been mitigated.

7. resolved Aug 22, 2026, 02:03 PM UTC

   Description: The issue with Firebase Performance Monitoring has been resolved for all affected users as of Saturday, 2026-08-22 06:50 PDT. We thank you for your patience while we worked on resolving the issue. Diagnosis / customer symptoms: The Firebase Dashboard has been fixed and currently is up to date, BigQuery was delayed for 2026-08-18 and 2026-08-19. All other Performance Monitoring data and BigQuery exports remained accurate. There was no data loss associated with this event. Workaround: None as the issue has been mitigated.

Read the full incident report →

Started Aug 18, 2026, 11:58 PM UTC· resolved Aug 19, 2026, 12:40 PM UTC

1. monitoring Aug 19, 2026, 03:57 AM UTC

   Description: We are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT. Our engineering team is currently working on a potential fix while continuing to investigate the full root cause. We will provide an update by Tuesday, 2026-08-18 22:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Affected customers may experience slowness while using Google Drive. Workaround: None at this time.

2. monitoring Aug 19, 2026, 04:59 AM UTC

   Description: We are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT. Our engineering team is currently working on a potential fix while continuing to investigate the full root cause. We will provide an update by Wednesday, 2026-08-19 00:30 PDT with current details. Symptoms: Affected customers in Australia and New Zealand may experience slowness while using Google Drive. Workaround: None at this time.

3. investigating Aug 19, 2026, 06:49 AM UTC

   Description: We are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT. Our engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution. We will provide an update by Wednesday, 2026-08-19 01:30 PDT with current details. Symptoms: Affected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive. Workaround: None at this time.

4. investigating Aug 19, 2026, 07:59 AM UTC

   Description: We are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT. Our engineering team has identified the root cause of the issue and is actively working on a mitigation strategy. There is no ETA for full resolution. We will provide an update by Wednesday, 2026-08-19 02:00 PDT with current details. Symptoms: Affected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive. Workaround: None at this time.

5. investigating Aug 19, 2026, 08:54 AM UTC

   Description: We are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT. Our engineering team is actively working on a mitigation strategy. There is no ETA for full resolution. We will provide an update by Wednesday, 2026-08-19 03:00 PDT with current details. Symptoms: Affected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive. Workaround: None at this time.

6. investigating Aug 19, 2026, 09:54 AM UTC

   Description: We are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT. Our engineering team is actively working on a mitigation strategy. There is no ETA for full resolution. We will provide an update by Wednesday, 2026-08-19 04:30 PDT with current details. Symptoms: Affected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive. Workaround: None at this time.

7. investigating Aug 19, 2026, 11:26 AM UTC

   Description: We are experiencing an issue with Google Drive beginning Tuesday, 2026-08-18 16:58 PDT. Our engineering team is actively working on a mitigation strategy. There is no ETA for full resolution. We will provide an update by Wednesday, 2026-08-19 06:00 PDT with current details. Symptoms: Affected customers in Australia and New Zealand may experience slowness or 502 errors, with some users briefly unable to access Drive. Workaround: None at this time.

8. resolved Aug 19, 2026, 01:05 PM UTC

   Description: The issue with Google Drive has been resolved for all affected customers as of Wednesday, 2026-08-19 05:40 PDT. Our engineering teams determined the slowness was the result of a localized capacity and routing issue within our regional infrastructure. To restore service, we implemented a fix to alleviate the network congestion and accelerated a planned infrastructure upgrade in the affected region to ensure normal performance levels. We thank you for your patience while we worked on resolving the issue. Symptoms: Affected customers in Australia and New Zealand may have experienced slowness or 502 errors, with some users briefly unable to access Drive. Workaround: The issue is now mitigated

9. resolved Aug 20, 2026, 08:00 PM UTC

   Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Workspace, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213 (https://support.google.com/a/answer/1047213). Date/Time of the Issue (All time US/Pacific) Incident Start: 18 August 2026 16:58 Incident End: 19 August 2026 05:40 Duration: 12 hours, 42 minutes Summary On August 18, 2026 from 16:58 to August 19, 2026 05:40 US/Pacific, Drive customers in Australia, Philippines and New Zealand experienced degraded performance and slowness. During this period users also experienced 502 errors. Other Google Workspace services were not affected. Preliminary Root Cause Google utilizes a multi-tiered global load balancing and network routing architecture to serve traffic efficiently. To ensure fair use and prevent congestion, bandwidth allocation is managed by attributing traffic to specific downstream services. A misconfiguration in the attribution of traffic caused the bandwidth usage to be incorrectly attributed to the frontend network load balancing tier itself, rather than to the Drive service. The sequence of events leading to customer impact was as follows: * Traffic from the regional load balancing tier was incorrectly attributed to the load balancing infrastructure rather than the Drive service. * Unrelated traffic caused network bandwidth usage attributed to the load balancing infrastructure in this region to exceed its quota. * Network bandwidth management systems applied throttling to this traffic to enforce quotas, causing severe network congestion affecting Drive traffic in this region. * This congestion led to increased latency, slowness, and error rates for users accessing Drive in the affected regions. Remediation Google engineers were alerted to the issue via automated monitoring and an influx of customer support cases, with initial incident response teams engaged by 17:21 US/Pacific on August 18, 2026. Engineers initially attempted to alleviate the congestion by draining regional frontend traffic to an alternate geographic location, but this proved ineffective and was reverted. The team then mitigated the issue by migrating regional traffic to an upgraded internal network routing architecture, which correctly set the traffic attribution for Drive. Normal operations were fully restored once 100% of the regional traffic was migrated to the upgraded network architecture. Description of Impact Between August 18, 2026, at 16:58 US/Pacific and August 19, 2026, at 05:40 US/Pacific, Google Drive users in Australia, Philippines and New Zealand experienced severe latency, degraded performance and 502 errors. During this roughly 12-hour and 42-minute window, affected customers may have observed significant slowness or errors when attempting to access Google Drive or listing files through Editors (e.g. docs.google.com (http://docs.google.com)), with initial load times significantly delayed. The issue was fully mitigated as of August 19, 2026, at 05:40 US/Pacific, and normal service has been restored.

10. resolved Aug 24, 2026, 04:31 PM UTC

    Google drive: * Elevated Latency: Users experienced extreme slowness when loading the Drive interface or accessing files, with initial data retrieval times heavily delayed. * Loading Failures and Errors: Many user requests timed out or returned HTTP 502 errors due to the aggressive network throttling dropping the connections. Scope exclusion: * Customers accessing Google Drive from outside the Australia, Philippines and New Zealand regions were largely unaffected by this localized throttling event. * Other Google Workspace applications not reliant on this specific routing path remained stable during the incident window.

Read the full incident report →

Started Aug 18, 2026, 07:03 PM UTC· resolved Aug 18, 2026, 10:30 PM UTC

1. monitoring Aug 18, 2026, 07:35 PM UTC

   Description: We are investigating increasing reports of access issues with Classroom’s new web homepage beginning on Thursday, 2026-08-13. Our engineering team is currently working on a potential fix while continuing to investigate the full root cause. We will provide an update by Tuesday, 2026-08-18 13:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms: Admins and some Teachers may experience permission errors when accessing Classroom’s new web homepage. Workaround: Users should still be able to navigate to other views from the left hand pane of the interface.

2. monitoring Aug 18, 2026, 08:00 PM UTC

   Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Tuesday, 2026-08-18 14:00 PDT. We will provide more information by Tuesday, 2026-08-18 14:15 PDT. Symptoms: Admins and some Teachers may experience permission errors when accessing Classroom’s new web homepage. Workaround: Users should still be able to navigate to other views from the left hand pane of the interface.

3. monitoring Aug 18, 2026, 08:27 PM UTC

   Description: Our engineering has identified the cause and mitigation work is currently underway. The mitigation rollout is expected to take several hours. We will provide more information by Wednesday, 2026-08-19 8:00 PDT. Symptoms: Workspace Admins may experience permission errors when accessing Classroom’s web homepage. Remaining Classroom functionality remains unaffected. The Classroom Mobile App and the Classroom API are unaffected by the issue. Workaround: All other pages are unaffected and admins can navigate to other pages using the left hand pane on the interface. Workspace Admins can still create classes without any issue. Workspace Admins can also view the courses that they are in by going to https://classroom.google.com/h/tv

4. resolved Aug 18, 2026, 11:02 PM UTC

   Description: The issue with Classroom has been resolved for all affected users as of Tuesday, 2026-08-18 15:30 PDT. From preliminary analysis, the issue was triggered due to a recent change which caused the permission mismatch, our Engineering team has rolled out the fix to mitigate the impact. We thank you for your patience while we worked on resolving the issue. Symptoms: Workspace Admins may have experienced permission errors when accessing Classroom’s web homepage. Remaining Classroom functionality remained unaffected. The Classroom Mobile App and the Classroom API were unaffected by the issue. Workaround: None as the issue is resolved

Read the full incident report →

Started Aug 18, 2026, 04:27 PM UTC· resolved Aug 21, 2026, 08:49 AM UTC

1. monitoring Aug 18, 2026, 04:28 PM UTC

   Released the August 2026 spam update , which applies globally and to all languages. The rollout may take a few days to complete.

2. resolved Aug 21, 2026, 08:49 AM UTC

   The rollout was complete as of August 21, 2026.

Read the full incident report →

Started Aug 17, 2026, 09:13 PM UTC· resolved Aug 17, 2026, 10:20 PM UTC

1. investigating Aug 17, 2026, 10:58 PM UTC

   App Distribution uploads started failing

2. resolved Aug 17, 2026, 10:59 PM UTC

   Incident has been resolved

Read the full incident report →

Started Aug 15, 2026, 12:24 AM UTC· resolved Aug 17, 2026, 11:04 PM UTC

1. monitoring Aug 15, 2026, 12:32 AM UTC

   The problem with Exchange Platforms has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. Some seller entries in the Seller.JSON file were missing domains for a period of a 2-3 days.

Read the full incident report →

Started Aug 13, 2026, 05:23 PM UTC· resolved Aug 14, 2026, 11:13 AM UTC

1. investigating Aug 13, 2026, 05:56 PM UTC

   We are investigating an issue with Firebase A/B Testing. Some users are not able to modify or create an experiment. We will provide more information shortly.

2. investigating Aug 13, 2026, 06:42 PM UTC

   We are still looking into this.

3. investigating Aug 13, 2026, 08:49 PM UTC

   We are still working on this. Meanwhile, please contact Firebase support for help in case you face any issues with A/B Testing

4. resolved Aug 14, 2026, 11:14 AM UTC

   The issue with Firebase A/B Testing has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Read the full incident report →

Started Aug 06, 2026, 01:06 PM UTC· resolved Aug 09, 2026, 03:02 PM UTC

1. resolved Aug 11, 2026, 11:48 AM UTC

   The problem with Content API for Shopping and Merchant Center has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Read the full incident report →

Started Aug 06, 2026, 01:06 PM UTC· resolved Aug 09, 2026, 03:12 PM UTC

1. investigating Aug 06, 2026, 05:46 PM UTC

   We're investigating reports of an issue with Merchant Center. We will provide more information shortly.

Read the full incident report →

Started Jul 31, 2026, 03:47 PM UTC· resolved Aug 01, 2026, 01:37 PM UTC

1. investigating Jul 31, 2026, 03:52 PM UTC

   We're aware of a problem with Google Ad Manager Data Transfer affecting a majority of users. We will provide an update by Jul 31, 2026, 5:47 PM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change. The affected users are able to access Google Ad Manager Data Transfer, but are seeing error messages, high latency, and/or other unexpected behavior. Jul 30, Hour 22 Google Ad Manager Data Transfer files are currently delayed for Reservation and Backfill File Types

2. investigating Jul 31, 2026, 05:23 PM UTC

   We are continuing to investigate this issue. We will provide an update by Jul 31, 2026, 7:00 PM UTC detailing when we expect to resolve the problem. The affected users are able to access Google Ad Manager Data Transfer, but are seeing error messages, high latency, and/or other unexpected behavior. Google Data Transfer Hour 18 (July 30) Reservation and Backfill files are delayed. Hour 20 (July 30) Backfill files are now published however, Reservation files remain delayed.

3. investigating Jul 31, 2026, 09:12 PM UTC

   We are continuing to investigate this issue. We will provide an update by Jul 31, 2026, 11:12 PM UTC detailing when we expect to resolve the problem. The affected users are able to access Google Ad Manager Data Transfer, but are seeing error messages, high latency, and/or other unexpected behavior. Google Data Transfer pipeline is still recovering but file publishing remains delayed.

4. resolved Aug 01, 2026, 01:38 PM UTC

   The problem with Google Ad Manager Data Transfer has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. The previously delayed Data Transfer Reservation and Backfill files are now successfully published. Processing delay is now back to normal.

Read the full incident report →

Started Jul 30, 2026, 09:00 PM UTC· resolved Jul 31, 2026, 10:20 AM UTC

1. investigating Jul 31, 2026, 03:38 AM UTC

   The engineering team is investigating elevated 501 "unimplemented" errors seen in Hosting's API when managing custom domains. It does not appear at this time that the data plane is effected.

2. investigating Jul 31, 2026, 04:38 AM UTC

   We believe to have found the root cause and are working to restore this portion of our API. We are more confident now that the data plane is unaffected and that this was limited in scope to the management of Hosting custom domains.

3. investigating Jul 31, 2026, 05:14 AM UTC

   The fix is being rolled out. The ETA for complete resolution is July 31 at ~11:00 UTC, though some users may have restored functionality before then.

Read the full incident report →

Started Jul 22, 2026, 11:06 AM UTC· resolved Jul 27, 2026, 03:23 PM UTC

1. investigating Jul 23, 2026, 05:44 AM UTC

   We're investigating reports of an issue with Play Subscriptions Renewal and Management. We will provide more information shortly. Most users are experiencing error messages when managing a subscription.

2. resolved Jul 27, 2026, 07:22 PM UTC

   The problem with Play Subscriptions Renewal and Management has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Read the full incident report →

Started Jul 21, 2026, 06:09 PM UTC· resolved Jul 24, 2026, 04:00 PM UTC

1. investigating Jul 21, 2026, 06:11 PM UTC

   Firebase Crashlytics data is delayed

Read the full incident report →

Started Jul 20, 2026, 03:38 PM UTC· resolved Jul 21, 2026, 04:20 PM UTC

1. investigating Jul 20, 2026, 03:39 PM UTC

   Firebase Test Lab is experiencing high execution latency for arm devices. We are currently investigating the cause and will provide more information shortly.

2. investigating Jul 20, 2026, 04:56 PM UTC

   Firebase Test Lab is currently experiencing increased execution error rates for arm and physical devices (Android and iOS). We have investigated the cause and are mitigating the issue.

3. investigating Jul 21, 2026, 02:31 PM UTC

   Firebase Test Lab is experiencing high execution latency for arm devices. We are currently investigating the cause and will provide more information shortly.

Read the full incident report →

Started Jul 15, 2026, 11:57 PM UTC· resolved Jul 16, 2026, 12:25 PM UTC

1. investigating Jul 16, 2026, 03:30 AM UTC

   Description: GCVE: Our engineering teams are currently working with data center facility teams to address the rising temperatures. A shutdown has already occurred for some private clouds, and we expect that others will shut down as well. We recommend shutting down your VMware workload VMs for precaution so that recovery is safer. A shutdown of all systems started at 19:00 PDT [4:00 Local Time (GMT+2)]. We will notify you as soon as your Private Cloud has been powered back up and it is safe to restart the VMs. BMS: Some Bare Metal Solution servers are experiencing high temperatures and we are in the process of shutting them down. Customers may be unable to access their systems until we resolve the issue. Google Cloud NetApp Volumes Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 21:30 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: GCVE: Customers may have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and a recovery, we are shutting down all Private Clouds in this zone. BMS: Customers may have already lost connectivity to their BMS machines. As a protective measure for hardware integrity and a recovery, we are shutting down servers in this zone. Google Cloud NetApp Volumes Customers may have already lost connectivity to ONTAP clusters, or will lose connectivity to them shortly, as we proactively shut them down for their protection. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

2. investigating Jul 16, 2026, 04:17 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google has proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. The datacenter operator team is working diligently to bring cooling and power back online so we can safely restart affected workloads. GCVE: Our engineering teams have shut down private clouds, and we expect that others will shut down as well. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. BMS: Some Bare Metal Solution servers are experiencing high temperatures and we are in the process of shutting them down. Customers may be unable to access their systems until we resolve the issue. Google Cloud NetApp Volumes Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 22:30 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and a recovery, we have shut down all private clouds in this zone. BMS: Customers may have already lost connectivity to their BMS machines. As a protective measure for hardware integrity and a recovery, we are shutting down some servers in this zone. Google Cloud NetApp Volumes Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down for their protection. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

3. investigating Jul 16, 2026, 04:41 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google has proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. The datacenter operator team is working diligently to bring cooling and power back online so we can safely restart affected workloads. Some cooling has been restored, and we are working to restore affected workloads as available capacity allows. GCVE: Our engineering teams have shut down private clouds to protect these clouds from any damage. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. BMS: Some Bare Metal Solution servers experienced high temperatures and we proactively shut them down. Customers may have been unable to access their systems, but impacted systems have now been restored. Google Cloud NetApp Volumes: Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 22:45 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. BMS: Customers may have lost connectivity to their BMS machines. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone, but the impacted servers have been restarted and should be functioning normally. Google Cloud NetApp Volumes: Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

4. investigating Jul 16, 2026, 06:29 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now working to safely restart affected workloads. GCVE: As a precautionary measure, our engineering teams had shut down private clouds to protect these clouds from any damage. Service restoration is still in progress. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. BMS: We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Our engineering team had proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. We will provide an update by Thursday, 2026-07-16 00:30 PDT with current details. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone. We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

5. investigating Jul 16, 2026, 07:39 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now working to safely restart affected workloads. GCVE: As a precautionary measure, our engineering teams had shut down private clouds to protect these clouds from any damage. Network fabric has been restored, and Private Cloud restoration remains underway. We will notify affected customers once their Private Clouds are restored to allow for VM operational verification. BMS: We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Our engineering team had proactively shut down ONTAP clusters to protect the data stored on them. Recovery is currently in progress. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. We will provide an update by Thursday, 2026-07-16 01:30 PDT with current details. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone. We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

6. investigating Jul 16, 2026, 08:32 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now validating site stability. GCVE: Network fabric has been restored, and Private Cloud restoration remains underway. We are notifying the affected customers as their Private Clouds are restored to allow for VM operational verification. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. Google Cloud NetApp Volumes: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide an update by Thursday, 2026-07-16 02:30 PDT with current details. Customer symptoms: GCVE: Customers would have observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we had shut down all Private Clouds in this zone. We are in the process of bringing all restored Private Clouds online. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Google Cloud NetApp Volumes: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. Workaround: There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

7. investigating Jul 16, 2026, 09:21 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now validating site stability. GCVE: Network fabric has been restored, and Private Cloud restoration remains underway. We are notifying the affected customers as their Private Clouds are restored to allow for VM operational verification. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. Google Cloud NetApp Volumes: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide an update by Thursday, 2026-07-16 03:30 PDT with current details. Customer symptoms: GCVE: Customers would have observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we had shut down all Private Clouds in this zone. We are in the process of bringing all restored Private Clouds online. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Google Cloud NetApp Volumes: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. Workaround: None at this time.

8. investigating Jul 16, 2026, 10:14 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. Google Cloud NetApp Volumes [Mitigated]: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. For Google Cloud NetApp Volumes and GCVE, if customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide the BMS restoration status update by Thursday, 2026-07-16 04:30 PDT with current details. Customer symptoms: The issue is now resolved for GCVE and Google Cloud NetApp Volumes. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Customers would have observed a loss of connectivity to their private cloud. Google Cloud NetApp Volumes [Mitigated]: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Workaround: None at this time.

9. investigating Jul 16, 2026, 11:39 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. Google Cloud NetApp Volumes [Mitigated]: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. For Google Cloud NetApp Volumes and GCVE, if customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide the BMS restoration status update by Thursday, 2026-07-16 05:30 PDT with current details. Customer symptoms: The issue is now resolved for GCVE and Google Cloud NetApp Volumes. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Customers would have observed a loss of connectivity to their private cloud. Google Cloud NetApp Volumes [Mitigated]: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Workaround: None at this time.

10. resolved Jul 16, 2026, 12:25 PM UTC

    Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users as of Thursday, 2026-07-16 02:31 PDT and 01:00 PDT respectively. The issue with BMS is largely resolved and is believed to be affecting a very small number of customers. Our Engineering will continue to recover this residual impact directly with the impacted customers. If you have questions or are still impacted, please open a case with the Support Team and we will work with you until this issue is resolved. GCVE: Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. Google Cloud NetApp Volumes: Service restoration is complete, and all affected servers are now operational. BMS: Service restoration is largely complete. We continue working with a very small number of remaining affected customers to achieve full recovery. We thank you for your patience while we continue to fully resolve the issue. Customer symptoms: The issue is now resolved for GCVE and Google Cloud NetApp Volumes. The issue with BMS is largely resolved and our engineering will continue to recover this residual impact directly with the impacted customers. GCVE: Customers would have observed a loss of connectivity to their private cloud. Google Cloud NetApp Volumes: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. Workaround: None at this time.

11. resolved Jul 18, 2026, 12:22 AM UTC

    Google cloud vmware engine impact: - Start: 15 July 2026 17:09 - End: 16 July 2026 2:33 - Duration: 9 Hours, 24 Minutes Google cloud netapp volumes impact: - Start: 15 July 2026 16:39 - End: 16 July 2026 01:10 - Duration: 8 Hours 31 Minutes Bare metal solution impact: - Start: 15 July 2026 18:37 - End: 16 July 2026 07:34 - Duration: 12 Hours 57 Minutes Summary On Wednesday, 15 July 2026, Google Cloud VMware Engine, Bare Metal Solution, and Google Cloud NetApp Volumes experienced service interruptions for a total duration of 14 hours, 55 minutes. We are taking immediate steps to ensure this doesn’t happen again. Preliminary Root Cause A regional data center hosting services in europe-west4-a experienced a loss of utility power and subsequent cooling capacity. The sequence of events leading to customer impact was as follows: - An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution gear and cooling equipment. - The loss of cooling infrastructure resulted in ambient temperatures rising rapidly within the affected data halls. Host servers, storage clusters, and network switches were shut down to prevent equipment damages due to the extreme heat. - The high temperatures in the data hall resulted in disruption to customer workloads and control plane operations across the affected services. Google engineers have begun a full root cause analysis and we will provide additional information once it is available. Remediation Google engineering teams were alerted to the issue via automated temperature and hardware unreachability telemetry starting at 16:39 US/Pacific. Engineers collaborated with the third-party facility provider to safely restore primary utility power and cooling systems, returning ambient room temperatures to safe operational levels. With the environment stabilized, field support engineers and remote teams systematically booted and verified the server hosts, storage nodes, and network fabric in a controlled sequence. Automated recovery playbooks were executed to restore underlying network switches and routing infrastructure, allowing storage appliances and compute clusters to be brought back online and verified for health. The underlying infrastructure has been fully recovered, and normal operations have been successfully restored. Description of Impact On 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone: - Google Cloud VMware Engine: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection. - Bare Metal Solution: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down. - Google Cloud NetApp Volumes: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.

12. resolved Jul 25, 2026, 01:16 PM UTC

    Incident Report Summary On Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes. The root cause for this outage is a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action. This incident was mitigated by the data center provider rectifying the failed systems followed by the Google engineering team restoring the services. Root Cause A regional data center hosting services in europe-west4-a experienced an upstream voltage transient affecting both utility power feeds A and B. During this event, the utility breakers on both A and B feeds tripped and initiated the transfer to the back-up power source DRUPS (Diesel Rotary Uninterruptible Power Supply). The transition of side B to DRUPS system was successful without any power interruption. The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures. The DRUPS failure to take over load initiated automatic transfer of the affected 3 rows from feed A to redundant power feed B. Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and experienced complete power loss due to an overload protection breaker trip. The root cause of the failed transfer on the affected row was identified as load deployment discrepancy which is under further investigation by Google engineering team. This resulted in loss of both redundant power feeds to the single row. During the voltage transient event, the server data hall experienced an increased temperature due to a cooling system failure. The chiller controller dropped offline during the voltage transient event, failing to signal the chilled water distribution pumps to restart and ultimately causing the chiller system A to shut down. The redundant source was not available due to known ongoing construction work at the facility. This resulted in the data hall temperatures reaching 44°C and subsequent shutdown of the affected data hall machines. Google engineering teams initiated machine shutdown procedures for the remainder of reachable devices as part of the cooling emergency shutdown process. A timeline of events during the incident is provided below. DataCenter Events | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:24 | An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution. | | 07-15-2026 16:24 | DRUPS back-up power system for side A failed to take over the data hall load. | | 07-15-2026 16:24 | The chiller controller dropped offline during the voltage transient event causing distribution pumps to stop and unable to restart. | | 07-15-2026 16:37 | Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and lost power. | | 07-15-2026 18:29 | Data hall temperatures reached 44C and crossed the safe machine operating threshold. | | 07-15-2026 19:55 | Machine shutdown procedures implemented. | | 07-15-2026 21:05 | Cooling system fully recovered and temperatures in the data hall returned to normal operating range. | | 07-15-2026 21:24 | Notification about cooling recovery sent by provider | | 07-15-2026 21:46 | Notification about power recovery sent by provider | GCVE Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:30 | GCVE service power monitoring alert received | | 07-15-2026 16:41 | First symptom detected, servers reporting redundancy power feed failure. | | 07-15-2026 17:05 | Switch temperature >60°C alerts triggered. | | 07-15-2026 17:09 | Start of user impact, first prober alert failure for north south traffic for customers. | | 07-15-2026 19:00 | Server / Network devices shutdown initiated. | | 07-15-2026 19:55 | All reachable server/network devices were shut down. | | 07-15-2026 21:24 | Cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 21:56 | Network restoration started with reachable hydra devices. | | 07-15-2026 22:40 | Onsite technician arrived, to recover the console servers. | | 07-15-2026 23:10 | Console server reboot/recovery complete. | | 07-16-2026 00:33 | Network recovery for all Placement Groups complete. | | 07-16-2026 02:36 | Incident mitigated, after recovering all the customer PCs are fully healthy. | This outage impacted 24 private clouds of 20 distinct customers in europe-west4-a region. Bare Metal Solutions (BMS) Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:41 | First symptom detected, servers and storage reporting redundancy power feed failure | | 07-15-2026 17:52 | Automated monitoring detected rising temperatures | | 07-15-2026 18:37 | Four Netapp SAN storage nodes failed and were shut down due to overheating, resulting in storage availability issue | | 07-15-2026 18:37 | Customer impact started, first server shutdown detected | | 07-15-2026 18:59 | Reserve and buffer servers started to get powered off | | 07-15-2026 20:43 | Drop in the temperature has been detected | | 07-15-2026 20:52 | Four Netapp SAN storage nodes recovered, mitigating storage availability issue | | 07-15-2026 21:05 | The cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 23:23 | First customer server reboot started | | 07-16-2026 06:32 | Last customer server reboot started | | 07-16-2026 08:42 | Incident mitigated. All impacted customer servers were rebooted and confirmed in healthy state. | This outage impacted 9 distinct BMS customers in europe-west4 NetApp Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:32 | Netapp facility remote monitoring alert on high temp and network switches switches failure received | | 07-15-2026 17:32 | All 6 clusters failed and were shut down automatically due to high temperature | | 07-15-2026 21:05 | Cooling capacity restored | | 07-15-2026 22:08 | Cluster recovery process initiated for all 6 clusters. 1st cluster recovered | | 07-15-2026 23:55 | All 6 Netapp Clusters and all customers recovered and validated as healthy & operational | Remediation and Prevention Google engineers were alerted to the infrastructure power loss on Wednesday, 15 July 2026 at 16:30 US/Pacific and subsequent increase in data hall temperatures at 17:44 US/Pacific via our automated monitoring and telemetry, and immediately began investigating. During the voltage transient event, the chillers maintained power but shut down because the distribution pumps failed to restart, resulting in no water flow. The pumps were manually switched from auto to hand mode to restore circulation and cooling. The facility team deployed an interim portable UPS to support the chiller controllers and prevent localized controller power loss. Upon confirmation of utility power restoration, the facility returned to utility power via automatic switching with the exception of two Remote Power Panels (RPPs), which required manual verification before restoration of power to affected server row. To provide full resiliency the facility team aligned a swing DRUPS unit to restore the redundant configuration on the A feed and is working on repairing the faulty DRUPS. To remediate the power loss at Row 3, once the electrical fault was confirmed to be fully isolated, teams reset the tripped breakers, successfully restoring both redundant power to the impacted server racks. Google is committed to preventing a repeat of this issue in the future and is completing the following actions: * Detailed investigation by the engineering team of the sequence of events that prevented transfer of load to DRUPS system and identified system improvements [ETA Aug 2026]. * Determine and resolve the cause of overload conditions that caused power loss to Row 3 [ETA Aug 2026]. * Perform investigation of chiller pump control system redundancy setup to ensure cooling system resiliency [ETA Sep 2026]. * Implement improvements to facility system monitoring alerts for power events and configure early alert thresholds for cooling excursion detection. [ETA Aug 2026]. * The GCVE engineering team is working to improve the automated triggering of shutdown during quickly evolving thermal runoff situations [ETA Oct 2026]. * The BMS engineering team is working with partner teams to update the incident classification and SLO definitions to improve personnel availability during major datacenter events [ETA Sep 2026]. * The GCNV engineering team is collaborate with the data center team to review incident handling SLA and runbooks, identify potential future occurrences and create playbooks to address/mitigate them, establish monthly joint emergency drill, and review if the Netapp cluster recovery process can be expedited [ETA Aug 2026]. Detailed Description of Impact On 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone: * Google Cloud VMware Engine: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection. * Bare Metal Solution: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down. * Google Cloud NetApp Volumes: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.

Read the full incident report →

Started Jul 15, 2026, 11:57 PM UTC· resolved Jul 16, 2026, 12:25 PM UTC

1. investigating Jul 16, 2026, 03:30 AM UTC

   Description: GCVE: Our engineering teams are currently working with data center facility teams to address the rising temperatures. A shutdown has already occurred for some private clouds, and we expect that others will shut down as well. We recommend shutting down your VMware workload VMs for precaution so that recovery is safer. A shutdown of all systems started at 19:00 PDT [4:00 Local Time (GMT+2)]. We will notify you as soon as your Private Cloud has been powered back up and it is safe to restart the VMs. BMS: Some Bare Metal Solution servers are experiencing high temperatures and we are in the process of shutting them down. Customers may be unable to access their systems until we resolve the issue. Google Cloud NetApp Volumes Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 21:30 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: GCVE: Customers may have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and a recovery, we are shutting down all Private Clouds in this zone. BMS: Customers may have already lost connectivity to their BMS machines. As a protective measure for hardware integrity and a recovery, we are shutting down servers in this zone. Google Cloud NetApp Volumes Customers may have already lost connectivity to ONTAP clusters, or will lose connectivity to them shortly, as we proactively shut them down for their protection. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

2. investigating Jul 16, 2026, 04:17 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google has proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. The datacenter operator team is working diligently to bring cooling and power back online so we can safely restart affected workloads. GCVE: Our engineering teams have shut down private clouds, and we expect that others will shut down as well. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. BMS: Some Bare Metal Solution servers are experiencing high temperatures and we are in the process of shutting them down. Customers may be unable to access their systems until we resolve the issue. Google Cloud NetApp Volumes Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 22:30 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and a recovery, we have shut down all private clouds in this zone. BMS: Customers may have already lost connectivity to their BMS machines. As a protective measure for hardware integrity and a recovery, we are shutting down some servers in this zone. Google Cloud NetApp Volumes Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down for their protection. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

3. investigating Jul 16, 2026, 04:41 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google has proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. The datacenter operator team is working diligently to bring cooling and power back online so we can safely restart affected workloads. Some cooling has been restored, and we are working to restore affected workloads as available capacity allows. GCVE: Our engineering teams have shut down private clouds to protect these clouds from any damage. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. BMS: Some Bare Metal Solution servers experienced high temperatures and we proactively shut them down. Customers may have been unable to access their systems, but impacted systems have now been restored. Google Cloud NetApp Volumes: Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 22:45 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. BMS: Customers may have lost connectivity to their BMS machines. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone, but the impacted servers have been restarted and should be functioning normally. Google Cloud NetApp Volumes: Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

4. investigating Jul 16, 2026, 06:29 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now working to safely restart affected workloads. GCVE: As a precautionary measure, our engineering teams had shut down private clouds to protect these clouds from any damage. Service restoration is still in progress. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. BMS: We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Our engineering team had proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. We will provide an update by Thursday, 2026-07-16 00:30 PDT with current details. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone. We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

5. investigating Jul 16, 2026, 07:39 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now working to safely restart affected workloads. GCVE: As a precautionary measure, our engineering teams had shut down private clouds to protect these clouds from any damage. Network fabric has been restored, and Private Cloud restoration remains underway. We will notify affected customers once their Private Clouds are restored to allow for VM operational verification. BMS: We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Our engineering team had proactively shut down ONTAP clusters to protect the data stored on them. Recovery is currently in progress. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. We will provide an update by Thursday, 2026-07-16 01:30 PDT with current details. Customer symptoms: GCVE: Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone. We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. Google Cloud NetApp Volumes: Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. Workaround: There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

6. investigating Jul 16, 2026, 08:32 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now validating site stability. GCVE: Network fabric has been restored, and Private Cloud restoration remains underway. We are notifying the affected customers as their Private Clouds are restored to allow for VM operational verification. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. Google Cloud NetApp Volumes: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide an update by Thursday, 2026-07-16 02:30 PDT with current details. Customer symptoms: GCVE: Customers would have observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we had shut down all Private Clouds in this zone. We are in the process of bringing all restored Private Clouds online. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Google Cloud NetApp Volumes: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. Workaround: There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

7. investigating Jul 16, 2026, 09:21 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now validating site stability. GCVE: Network fabric has been restored, and Private Cloud restoration remains underway. We are notifying the affected customers as their Private Clouds are restored to allow for VM operational verification. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. Google Cloud NetApp Volumes: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide an update by Thursday, 2026-07-16 03:30 PDT with current details. Customer symptoms: GCVE: Customers would have observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we had shut down all Private Clouds in this zone. We are in the process of bringing all restored Private Clouds online. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Google Cloud NetApp Volumes: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. Workaround: None at this time.

8. investigating Jul 16, 2026, 10:14 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. Google Cloud NetApp Volumes [Mitigated]: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. For Google Cloud NetApp Volumes and GCVE, if customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide the BMS restoration status update by Thursday, 2026-07-16 04:30 PDT with current details. Customer symptoms: The issue is now resolved for GCVE and Google Cloud NetApp Volumes. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Customers would have observed a loss of connectivity to their private cloud. Google Cloud NetApp Volumes [Mitigated]: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Workaround: None at this time.

9. investigating Jul 16, 2026, 11:39 AM UTC

   Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. Google Cloud NetApp Volumes [Mitigated]: Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. BMS: Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. For Google Cloud NetApp Volumes and GCVE, if customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide the BMS restoration status update by Thursday, 2026-07-16 05:30 PDT with current details. Customer symptoms: The issue is now resolved for GCVE and Google Cloud NetApp Volumes. We continue to recover the impact for BMS at the moment. GCVE [Mitigated]: Customers would have observed a loss of connectivity to their private cloud. Google Cloud NetApp Volumes [Mitigated]: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. Workaround: None at this time.

10. resolved Jul 16, 2026, 12:25 PM UTC

    Description: The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users as of Thursday, 2026-07-16 02:31 PDT and 01:00 PDT respectively. The issue with BMS is largely resolved and is believed to be affecting a very small number of customers. Our Engineering will continue to recover this residual impact directly with the impacted customers. If you have questions or are still impacted, please open a case with the Support Team and we will work with you until this issue is resolved. GCVE: Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. Google Cloud NetApp Volumes: Service restoration is complete, and all affected servers are now operational. BMS: Service restoration is largely complete. We continue working with a very small number of remaining affected customers to achieve full recovery. We thank you for your patience while we continue to fully resolve the issue. Customer symptoms: The issue is now resolved for GCVE and Google Cloud NetApp Volumes. The issue with BMS is largely resolved and our engineering will continue to recover this residual impact directly with the impacted customers. GCVE: Customers would have observed a loss of connectivity to their private cloud. Google Cloud NetApp Volumes: Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. BMS: Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. Workaround: None at this time.

11. resolved Jul 18, 2026, 12:22 AM UTC

    Google cloud vmware engine impact: - Start: 15 July 2026 17:09 - End: 16 July 2026 2:33 - Duration: 9 Hours, 24 Minutes Google cloud netapp volumes impact: - Start: 15 July 2026 16:39 - End: 16 July 2026 01:10 - Duration: 8 Hours 31 Minutes Bare metal solution impact: - Start: 15 July 2026 18:37 - End: 16 July 2026 07:34 - Duration: 12 Hours 57 Minutes Summary On Wednesday, 15 July 2026, Google Cloud VMware Engine, Bare Metal Solution, and Google Cloud NetApp Volumes experienced service interruptions for a total duration of 14 hours, 55 minutes. We are taking immediate steps to ensure this doesn’t happen again. Preliminary Root Cause A regional data center hosting services in europe-west4-a experienced a loss of utility power and subsequent cooling capacity. The sequence of events leading to customer impact was as follows: - An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution gear and cooling equipment. - The loss of cooling infrastructure resulted in ambient temperatures rising rapidly within the affected data halls. Host servers, storage clusters, and network switches were shut down to prevent equipment damages due to the extreme heat. - The high temperatures in the data hall resulted in disruption to customer workloads and control plane operations across the affected services. Google engineers have begun a full root cause analysis and we will provide additional information once it is available. Remediation Google engineering teams were alerted to the issue via automated temperature and hardware unreachability telemetry starting at 16:39 US/Pacific. Engineers collaborated with the third-party facility provider to safely restore primary utility power and cooling systems, returning ambient room temperatures to safe operational levels. With the environment stabilized, field support engineers and remote teams systematically booted and verified the server hosts, storage nodes, and network fabric in a controlled sequence. Automated recovery playbooks were executed to restore underlying network switches and routing infrastructure, allowing storage appliances and compute clusters to be brought back online and verified for health. The underlying infrastructure has been fully recovered, and normal operations have been successfully restored. Description of Impact On 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone: - Google Cloud VMware Engine: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection. - Bare Metal Solution: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down. - Google Cloud NetApp Volumes: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.

12. resolved Jul 25, 2026, 01:16 PM UTC

    Incident Report Summary On Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes. The root cause for this outage is a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action. This incident was mitigated by the data center provider rectifying the failed systems followed by the Google engineering team restoring the services. Root Cause A regional data center hosting services in europe-west4-a experienced an upstream voltage transient affecting both utility power feeds A and B. During this event, the utility breakers on both A and B feeds tripped and initiated the transfer to the back-up power source DRUPS (Diesel Rotary Uninterruptible Power Supply). The transition of side B to DRUPS system was successful without any power interruption. The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures. The DRUPS failure to take over load initiated automatic transfer of the affected 3 rows from feed A to redundant power feed B. Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and experienced complete power loss due to an overload protection breaker trip. The root cause of the failed transfer on the affected row was identified as load deployment discrepancy which is under further investigation by Google engineering team. This resulted in loss of both redundant power feeds to the single row. During the voltage transient event, the server data hall experienced an increased temperature due to a cooling system failure. The chiller controller dropped offline during the voltage transient event, failing to signal the chilled water distribution pumps to restart and ultimately causing the chiller system A to shut down. The redundant source was not available due to known ongoing construction work at the facility. This resulted in the data hall temperatures reaching 44°C and subsequent shutdown of the affected data hall machines. Google engineering teams initiated machine shutdown procedures for the remainder of reachable devices as part of the cooling emergency shutdown process. A timeline of events during the incident is provided below. DataCenter Events | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:24 | An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution. | | 07-15-2026 16:24 | DRUPS back-up power system for side A failed to take over the data hall load. | | 07-15-2026 16:24 | The chiller controller dropped offline during the voltage transient event causing distribution pumps to stop and unable to restart. | | 07-15-2026 16:37 | Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and lost power. | | 07-15-2026 18:29 | Data hall temperatures reached 44C and crossed the safe machine operating threshold. | | 07-15-2026 19:55 | Machine shutdown procedures implemented. | | 07-15-2026 21:05 | Cooling system fully recovered and temperatures in the data hall returned to normal operating range. | | 07-15-2026 21:24 | Notification about cooling recovery sent by provider | | 07-15-2026 21:46 | Notification about power recovery sent by provider | GCVE Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:30 | GCVE service power monitoring alert received | | 07-15-2026 16:41 | First symptom detected, servers reporting redundancy power feed failure. | | 07-15-2026 17:05 | Switch temperature >60°C alerts triggered. | | 07-15-2026 17:09 | Start of user impact, first prober alert failure for north south traffic for customers. | | 07-15-2026 19:00 | Server / Network devices shutdown initiated. | | 07-15-2026 19:55 | All reachable server/network devices were shut down. | | 07-15-2026 21:24 | Cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 21:56 | Network restoration started with reachable hydra devices. | | 07-15-2026 22:40 | Onsite technician arrived, to recover the console servers. | | 07-15-2026 23:10 | Console server reboot/recovery complete. | | 07-16-2026 00:33 | Network recovery for all Placement Groups complete. | | 07-16-2026 02:36 | Incident mitigated, after recovering all the customer PCs are fully healthy. | This outage impacted 24 private clouds of 20 distinct customers in europe-west4-a region. Bare Metal Solutions (BMS) Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:41 | First symptom detected, servers and storage reporting redundancy power feed failure | | 07-15-2026 17:52 | Automated monitoring detected rising temperatures | | 07-15-2026 18:37 | Four Netapp SAN storage nodes failed and were shut down due to overheating, resulting in storage availability issue | | 07-15-2026 18:37 | Customer impact started, first server shutdown detected | | 07-15-2026 18:59 | Reserve and buffer servers started to get powered off | | 07-15-2026 20:43 | Drop in the temperature has been detected | | 07-15-2026 20:52 | Four Netapp SAN storage nodes recovered, mitigating storage availability issue | | 07-15-2026 21:05 | The cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 23:23 | First customer server reboot started | | 07-16-2026 06:32 | Last customer server reboot started | | 07-16-2026 08:42 | Incident mitigated. All impacted customer servers were rebooted and confirmed in healthy state. | This outage impacted 9 distinct BMS customers in europe-west4 NetApp Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:32 | Netapp facility remote monitoring alert on high temp and network switches switches failure received | | 07-15-2026 17:32 | All 6 clusters failed and were shut down automatically due to high temperature | | 07-15-2026 21:05 | Cooling capacity restored | | 07-15-2026 22:08 | Cluster recovery process initiated for all 6 clusters. 1st cluster recovered | | 07-15-2026 23:55 | All 6 Netapp Clusters and all customers recovered and validated as healthy & operational | Remediation and Prevention Google engineers were alerted to the infrastructure power loss on Wednesday, 15 July 2026 at 16:30 US/Pacific and subsequent increase in data hall temperatures at 17:44 US/Pacific via our automated monitoring and telemetry, and immediately began investigating. During the voltage transient event, the chillers maintained power but shut down because the distribution pumps failed to restart, resulting in no water flow. The pumps were manually switched from auto to hand mode to restore circulation and cooling. The facility team deployed an interim portable UPS to support the chiller controllers and prevent localized controller power loss. Upon confirmation of utility power restoration, the facility returned to utility power via automatic switching with the exception of two Remote Power Panels (RPPs), which required manual verification before restoration of power to affected server row. To provide full resiliency the facility team aligned a swing DRUPS unit to restore the redundant configuration on the A feed and is working on repairing the faulty DRUPS. To remediate the power loss at Row 3, once the electrical fault was confirmed to be fully isolated, teams reset the tripped breakers, successfully restoring both redundant power to the impacted server racks. Google is committed to preventing a repeat of this issue in the future and is completing the following actions: * Detailed investigation by the engineering team of the sequence of events that prevented transfer of load to DRUPS system and identified system improvements [ETA Aug 2026]. * Determine and resolve the cause of overload conditions that caused power loss to Row 3 [ETA Aug 2026]. * Perform investigation of chiller pump control system redundancy setup to ensure cooling system resiliency [ETA Sep 2026]. * Implement improvements to facility system monitoring alerts for power events and configure early alert thresholds for cooling excursion detection. [ETA Aug 2026]. * The GCVE engineering team is working to improve the automated triggering of shutdown during quickly evolving thermal runoff situations [ETA Oct 2026]. * The BMS engineering team is working with partner teams to update the incident classification and SLO definitions to improve personnel availability during major datacenter events [ETA Sep 2026]. * The GCNV engineering team is collaborate with the data center team to review incident handling SLA and runbooks, identify potential future occurrences and create playbooks to address/mitigate them, establish monthly joint emergency drill, and review if the Netapp cluster recovery process can be expedited [ETA Aug 2026]. Detailed Description of Impact On 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone: * Google Cloud VMware Engine: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection. * Bare Metal Solution: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down. * Google Cloud NetApp Volumes: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.

Read the full incident report →