Google Workspace experienced a minor incident on August 25, 2025 affecting Google Voice, lasting 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 25, 2025, 08:01 PM UTC
Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Monday, 2025-08-25 15:00 PDT. Diagnosis / customer symptoms: Affected customers may have experienced dropped calls and connection issues intermittently. Workaround: None at this time.
- resolved Aug 25, 2025, 09:12 PM UTC
Description: The issue with Google Voice has been resolved for all affected users as of Monday, 2025-08-25 12:57 PDT The issue was due to a hardware device firmware update causing higher than expected traffic to an overloaded server. Our engineers have mitigated the issue by disabling the software change and adding extra capacity to the affected servers. We thank you for your patience while we worked on resolving the issue. Diagnosis / customer symptoms: Affected customers may have experienced dropped calls and connection issues intermittently. Workaround: None at this time.
- resolved Aug 26, 2025, 07:46 AM UTC
Affected services and features: Google Voice Regions/Zones: Global Description: Google Voice customers experienced intermittent dropped calls and connection issues globally for a duration of 4 hours. Based on preliminary analysis, the root cause is a software rollout which resulted in increased traffic and overloaded key infrastructure Google uses for voice calls. Customer impact: Affected customers intermittently experienced dropped calls and connection issues. This issue may have affected our enterprise and consumer account customers across all devices (mobile, web, deskphones or dedicated hardware).
- resolved Aug 29, 2025, 12:55 AM UTC
Incident Report Summary Incident Start: 25 August 2025 09:25 US/Pacific Incident End: 25 August 2025 13:25 US/Pacific Duration: 4 hours On Monday, 25 August 2025, Google Voice customers experienced intermittent dropped calls and connection issues globally for a duration of 4 hours. To our affected customers whose business was impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Root Cause A software update inadvertently caused a massive number of devices to simultaneously connect to a critical component of our voice infrastructure. This surge of connection attempts overwhelmed the component, resulting in server tasks dropping traffic and repeatedly crashing for approximately 4 hours. The issue was compounded because the devices were configured to immediately and repeatedly retry failed connections, creating a feedback loop that intensified the overload. As a result, this core component was unable to process traffic from other clients, including Google Voice. This resulted in intermittent dropped calls and a significant percentage of call failures during the incident’s peak. Remediation and Prevention Google engineers were alerted to the issue via an internal monitoring at 07:20 US/Pacific on 25 August and immediately started an investigation. Once the nature and scope of the issue was identified, Google engineers rolled back the software update and increased capacity to the affected servers.These actions successfully mitigated the issue for all affected users by 13:25 US/Pacific. Google is committed preventing a repeat of this issue in the future and is completing the following actions: - To prevent service degradation, client applications implement a longer retry backoff upon receiving an HTTP 503 error for connection failures. - Improve and update software rollout procedures and tooling to protect against unintentional changes and resulting traffic spikes. - Improve capacity planning and server utilization monitoring for the affected servers. We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future. Detailed Description of Impact From 09:25 to 13:25 US/Pacific on 25 August 2025, Google Voice customers experienced intermittent dropped calls and connection issues globally for a duration of 4 hours. Both enterprise and consumer users experienced intermittent dropped calls and connection issues across all devices such as mobile, web, and desk phones. At the peak of the incident, call failure rates reached up to 25%, with approximately one call dropped per second due to peer timeouts.