WellSaid Labs Outage History
WellSaid Labs is up right nowPingoru has recorded 19 incidents for WellSaid Labs: 3 in the last 12 months and 0 in the last 90 days.
Incidents, newest first
Minor July 24, 2026 Studio feature regression 34m Editor
Started Jul 24, 2026, 09:11 PM UTC· resolved Jul 24, 2026, 09:46 PM UTC
- investigating Jul 24, 2026, 09:11 PM UTC
We are investigating WellSaid Studio showing incorrect functionality to users.
- identified Jul 24, 2026, 09:13 PM UTC
The issue has been identified and the team and working on a resolution.
- resolved Jul 24, 2026, 09:46 PM UTC
The incident has been resolved and Studio functionality is back to normal operation.
Critical April 15, 2026 TTS Service Disruption –
Started Apr 15, 2026, 02:20 PM UTC· resolved Apr 15, 2026, 02:20 PM UTC
- resolved Apr 15, 2026, 02:20 PM UTC
We experienced a TTS service disruption at approximately 9 UTC. Our infrastructure team is worked to identify the root cause and implement a solution at approximately 5 UTC. Studio and API users were unable to generate new audio clips during that time. However Studio projects and existing audio were accessible.
Major February 4, 2026 Standard avatar partial outage –
Started Feb 04, 2026, 04:27 PM UTC· resolved Feb 04, 2026, 04:27 PM UTC
- resolved Feb 05, 2026, 05:38 PM UTC
Initial assessment: Starting at 1:55 AM PT there is an ongoing partial outage for Studio and API customer on "standard"/v10 avatar voices. Resolved 8:27AM PT.
Notice September 16, 2025 GPU Node Provisioning Failure 14d 8h Developer API
Started Sep 16, 2025, 03:00 PM UTC· resolved Sep 30, 2025, 11:16 PM UTC
- identified Sep 16, 2025, 04:25 PM UTC
We are currently experiencing issues provisioning new GPU nodes in our GKE environments. This is causing a reduced capacity in the number of requests we're able to serve simultaneously, leading to longer response times and clips failing to generate.
- monitoring Sep 17, 2025, 04:08 PM UTC
Provisioning of new GPU nodes in our GKE environments began returning to normal overnight. We are now seeing only minimal delays in node creation, and service performance has largely stabilized. We are working on expanding our infrastructure and are actively monitoring node availability.
- resolved Sep 30, 2025, 11:16 PM UTC
This incident has been resolved.
Major June 12, 2025 Cloud provider core infrastructure unavailability 2h 13m Editor, Developer API, API, Google Cloud Platform Google Kubernetes Engine and 3 more
Started Jun 12, 2025, 06:33 PM UTC· resolved Jun 12, 2025, 08:46 PM UTC
- investigating Jun 12, 2025, 06:33 PM UTC
We are currently experiencing issues with our Cloud provider's core infrastructure, which is affecting a large portion of network traffic. We are monitoring this outage at the PaloAltoNetworks level https://status.paloaltonetworks.com/incidents/l4zv6n67r8x6
- identified Jun 12, 2025, 06:33 PM UTC
The issue has been identified as outside our infrastructure and reported by our cloud providers.
- identified Jun 12, 2025, 06:40 PM UTC
We are continuing to work on a fix for this issue.
- identified Jun 12, 2025, 07:05 PM UTC
Google Cloud has reported a major service outage across IAM affecting the entire platform https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
- identified Jun 12, 2025, 07:45 PM UTC
Our cloud provider has reported that most locations have fully recovered. However, some features—such as clip interactions—remain unavailable. We’re actively working to restore full functionality and will provide further updates as they become available.
- monitoring Jun 12, 2025, 08:17 PM UTC
Services are beginning to recover, and clip creation is now fully operational. However, clips that were created during the outage remain unavailable. Our engineering team is actively investigating options to recover those affected clips.
- monitoring Jun 12, 2025, 08:32 PM UTC
Services are now operational, though some requests may still be slower than normal. Clip creation is functioning, but clips generated during the outage remain unavailable. Our team is continuing to monitor the system and investigating options to recover the affected clip data.
- resolved Jun 12, 2025, 08:46 PM UTC
Services are now fully operational, and all systems are performing as expected.
Critical May 20, 2025 Developer API Outage –
Started May 20, 2025, 08:30 AM UTC· resolved May 20, 2025, 08:30 AM UTC
- resolved May 21, 2025, 04:38 PM UTC
Service failure impacting all API customer traffic Incident Start: 2025-05-20 08:17:00 UTC Incident End: 2025-05-20 09:24:00 UTC Incident Duration: 67 minutes Impact: Complete service outage resulting in 502 errors for customer traffic
Minor April 22, 2025 Studio clip conversion/script downloading outage 11m API
Started Apr 22, 2025, 05:06 PM UTC· resolved Apr 22, 2025, 05:17 PM UTC
- identified Apr 22, 2025, 05:06 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 22, 2025, 05:14 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 22, 2025, 05:17 PM UTC
This incident has been resolved.
Minor November 10, 2024 Developer TTS API authorization failures 21m Developer API
Started Nov 10, 2024, 01:41 AM UTC· resolved Nov 10, 2024, 02:02 AM UTC
- identified Nov 10, 2024, 02:38 AM UTC
Some TTS Developer API nodes have started performing requests to internal TTS service with incorrect api keys.
- monitoring Nov 10, 2024, 02:41 AM UTC
Keys between the two services have been synced and we are seeing successful requests coming from the Developer API gateway. We are continuing to monitor to ensure all nodes reflect this change.
- resolved Nov 10, 2024, 02:44 AM UTC
All Developer TTS API gateway nodes are properly communicating with internal TTS service.
- postmortem Nov 10, 2024, 02:45 AM UTC
# Summary During a migration of our internal TTS service infrastructure, our external facing TTS Developer API experienced a partial outage due to authentication issues with the new TTS service. The root cause was identified as the use of invalid authentication keys from the Developer API which had not been updated to reflect those within the new internal TTS service. This resulted in a service degradation leading to 19 failed requests. # Timeline * 2024-11-10 01:39 UTC : Migration of internal TTS service to new infrastructure begins * 2024-11-10 01:41 UTC : Developer TTS API Gateway nodes begin to pick up new routing change * 2024-11-10 01:41 UTC : The nodes routing to the new TTS service begin to fail requests * 2024-11-10 01:42 UTC : Monitoring system shows failing requests between the two services * 2024-11-10 01:42 UTC : Routing logic changed back to existing infrastructure * 2024-11-10 01:43 UTC : Root cause of mismatching keys is identified * 2024-11-10 01:43 UTC : Monitoring system alert for failed requests to Developer TTS fires * 2024-11-10 01:44 UTC : Impact analysis begins * 2024-11-10 01:44 UTC : Developer TTS API Gateway successfully authenticates to existing internal TTS service * 2024-11-10 01:44 UTC : Existing authentication keys used by Developer TTS API Gateway are manually added to new TTS service * 2024-11-10 01:51 UTC : Verification of existing keys against new TTS infrastructure is performed * 2024-11-10 02:00 UTC : Impact analysis finds only Developer TTS API is affected * 2024-11-10 02:01 UTC : Routing logic changed back to new infrastructure * 2024-11-10 02:02 UTC : Developer TTS API Gateway begins to successfully authenticates to internal TTS service * 2024-11-10 02:05 UTC : Infrastructure team continues to monitor both services # Impact * Some users of the Developer TTS API hitting nodes connecting to the new TTS service would have experienced server errors while attempting to generate new clips, in total 19 requests over a 3 minute time frame failed # Action Items * A planned extension of this infrastructure migration is to implement automatic rotation and refreshing of API keys between the two services to remove the need for manual syncing