VMware Workspace ONE incident

Google Meet may be unavailable for users globally.

Major Resolved View vendor source →

VMware Workspace ONE experienced a major incident on September 26, 2025 affecting Google Meet, lasting 58m. The incident has been resolved; the full update timeline is below.

Started
Sep 26, 2025, 03:49 PM UTC
Resolved
Sep 26, 2025, 04:47 PM UTC
Duration
58m
Detected by Pingoru
Sep 26, 2025, 03:49 PM UTC

Affected components

Google Meet

Update timeline

  1. resolved Sep 26, 2025, 04:52 PM UTC

    Description: We are experiencing an issue with Google Meet beginning on Friday, 2025-09-26 08:49 PDT. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2025-09-26 11:00 PDT with current details. We apologize to all who are affected by the disruption. Customer symptoms: Users are unable to join calls on Google Meet. Workaround: None at this time.

  2. resolved Sep 26, 2025, 05:17 PM UTC

    Description: The issue with Google Meet has been resolved for all affected users as of Friday, 2025-09-26 09:47 PDT. A root cause analysis is in progress. Google will be publishing a preliminary Incident Report shortly and a full Incident Report in the coming days. We thank you for your patience while we worked on resolving the issue. Customer symptoms: Impacted users experienced errors when attempting to access Google Meet.

  3. resolved Sep 26, 2025, 07:08 PM UTC

    Affected services and features: Google Meet Regions/Zones: Global Description: Google Meet users experienced elevated error rates globally for a duration of 1 hour, 7 minutes. The preliminary root cause was a recent rollout to a backend API server, which was mitigated after the rollback completed. Google will complete a full Incident Report in the following days that will provide a full root cause. Customer impact: Affected users experienced 504 errors when attempting to access Google Meet. ---

  4. resolved Oct 02, 2025, 12:08 AM UTC

    Incident Report Summary On Friday, 26 September 2025 at 08:40 US/Pacific, Google Meet users experienced elevated error rates globally for 1 hour, 1 minute. To our Google Meet customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Root Cause The root cause of the incident was a flaw in the error-handling logic within the experimentation module's configuration evaluation. This flaw created the conditions for resource contention, where too many processes attempted to access the same system resource at once. The trigger was a specific change to the Meet experimentation configuration file, which caused an elevated error level during evaluation and led to the contention in the Meet backend. Our monitoring systems detected the issue right away, and engineers quickly identified the change as the source. They reverted the update, which resolved the issue. Remediation and Prevention Google engineers were alerted to the issue through the internal alerts at 08:49 US/Pacific on 26 September, 2025 and immediately started an investigation. They identified several recent changes as potential causes, and began systematically rolling back those changes. At 09:25 US/Pacific, engineers reverted the Meet API Server binary to a previous version. At 09:41 US/Pacific, engineers reverted and paused subsequent Meet backend API server configuration rollouts across all Google Meet production clusters thus mitigating the issue. At 11:52 US/Pacific, engineers reverted the change that caused lock contention. We sincerely apologize to our customers who were impacted by this disruption. This is not the level of quality and reliability we strive to offer, and we are taking the following steps to ensure this does not happen again. The Google Meet Engineering team is accelerating the deprecation of legacy code that caused lock contention. Detailed Description of Impact On Friday, 26 September 2025, from 08:40 to 09:41 US/Pacific, users were either unable to connect to Google Meet or experienced problems with establishing meetings. Users already in Google Meet sessions were likely not impacted. The issue affected up to 20 percent of Google Meet users globally.