VMware Workspace ONE incident
We're investigating reports of an issue with Gmail, Google Calendar, Google Chat, Google Cloud Search, Google Docs, Google Drive, Google Meet, Google Tasks, and Google Voice. We will provide more info
VMware Workspace ONE experienced a minor incident on June 12, 2025 affecting AppSheet and Gmail and 1 more component, lasting 5h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved Jun 12, 2025, 07:01 PM UTC
We're investigating reports of an issue with Gmail, Google Calendar, Google Chat, Google Cloud Search, Google Docs, Google Drive, Google Meet, Google Tasks, and Google Voice. We will provide more information shortly. Multiple Workspace products beginning on Thursday, at 2025-06-12 10:58 PDT may be experiencing service issues. Our engineers are currently investigating the issue. We apologize to all who are affected by the disruption.
- investigating Jun 12, 2025, 07:30 PM UTC
All product impacts except Google Meet have recovered. Google engineers continue to work on full mitigation.
- resolved Jun 12, 2025, 07:53 PM UTC
The issue with Google Chat, Google Meet, Gmail, Google Calendar, Google Drive, Google Cloud Search, Google Tasks, Google Voice has been resolved for all affected users as of Thursday, 2025-06-12 12:48 PDT. The issue with AppSheet has been resolved for all affected users as of Thursday, 2025-06-12 16:00 PDT. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue.
- resolved Jun 13, 2025, 07:00 AM UTC
(all times us/pacific): Incident Start: 12 June, 2025 10:49 All regions except us-central1 mitigated: 12 June, 2025 12:48 Incident End: 12 June, 2025 13:49 Duration: 3 hours Regions/Zones: Global Description: Multiple Google Cloud and Google Workspace products experienced increased 503 errors in external API requests, impacting customers. From our initial analysis, the issue occurred due to an invalid automated quota update to our API management system which was distributed globally, causing external API requests to be rejected. To recover we bypassed the offending quota check, which allowed recovery in most regions within 2 hours. However, the quota policy database in us-central1 became overloaded, resulting in much longer recovery in that region. Several products had moderate residual impact (e.g. backlogs) for up to an hour after the primary issue was mitigated and a small number recovering after that. Google will complete a full Incident Report in the following days that will provide a detailed root cause. Customer impact: Customers had intermittent API and user-interface access issues to the impacted services. Existing streaming and IaaS resources were not impacted. Additional details: This incident should not have happened, and we will take the following measures to prevent future recurrence: Prevent our API management platform from failing due to invalid or corrupt data. Prevent metadata from propagating globally without appropriate protection, testing and monitoring in place. Improve system error handling and comprehensive testing for handling of invalid data. Google cloud products: * Identity and Access Management * Cloud Build * Cloud Key Management Service * Google Cloud Storage * Cloud Monitoring * Google Cloud Dataproc * Cloud Security Command Center * Artifact Registry * Cloud Workflows * Cloud Healthcare * Resource Manager API * Dataproc Metastore * Cloud Run * VMWare engine * Dataplex * Migrate to Virtual Machines * Google BigQuery * Contact Center AI Platform * Google Cloud Deploy * Media CDN * Colab Enterprise * Vertex Gemini API * Cloud Data Fusion * Cloud Asset Inventory * Datastream * Integration Connectors * Apigee * Google Cloud NetApp Volumes * Google Cloud Bigtable * Looker (Google Cloud core) * Looker Studio * Google Cloud Functions * Cloud Load Balancing * Traffic Director * Document AI * AutoML Translation * Pub/Sub Lite * API Gateway * Agent Assist * AlloyDB for PostgreSQL * Cloud Firestore * Cloud Logging * Cloud Shell * Cloud Memorystore * Cloud Spanner * Contact Center Insights * Database Migration Service * Dialogflow CX * Dialogflow ES * Google App Engine * Google Cloud Composer * Google Cloud Console * Google Cloud DNS * Google Cloud Pub/Sub * Google Cloud SQL * Google Compute Engine * Identity Platform * Managed Service for Apache Kafka * Memorystore for Memcached * Memorystore for Redis * Memorystore for Redis Cluster * Persistent Disk * Personalized Service Health * Speech-to-Text * Text-to-Speech * Vertex AI Search * Retail API * Vertex AI Feature Store * BigQuery Data Transfer Service * Google Cloud Marketplace * Cloud NAT * Hybrid Connectivity * Cloud Vision * Network Connectivity Center * Cloud Workstations Google workspace products: * AppSheet * Gmail * Google Calendar * Google Drive * Google Chat * Google Voice * Google Docs * Google Meet * Google Cloud Search * Google Tasks
- resolved Jun 13, 2025, 11:59 PM UTC
The detailed Incident Report is now available on the Google Cloud Service Health Dashboard at https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW