Cartesia Outage History
Cartesia is up right nowPingoru has recorded 62 incidents for Cartesia: 56 in the last 12 months and 20 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 2 of 720 (0%) |
| May 2026 | 6 | 9 of 744 (1%) |
| June 2026 | 9 | 8 of 720 (1%) |
| July 2026 | 6 | 10 of 744 (1%) |
| August 2026 | 9 | 9 of 744 (1%) |
| September 2026 | 4 | 26 of 720 (4%) |
| October 2026 | 5 | 2 of 592 (0%) |
Incidents, newest first
Notice December 18, 2025 TTS API - US 22m Text to Speech (US)
Started Dec 18, 2025, 07:48 PM UTC· resolved Dec 18, 2025, 08:10 PM UTC
- identified Dec 18, 2025, 07:48 PM UTC
A faulty deploy temporarily took down our TTS on the US cluster. Traffic rerouted to backup clusters and auto rollback for US fired but there was a spike in 503 errors briefly. Looking into confirming this has been resolved
- monitoring Dec 18, 2025, 07:49 PM UTC
Healthchecks indicate this is currently working as intended, will continue to monitor for a bit before closing
- resolved Dec 18, 2025, 08:10 PM UTC
Confirmed there are no longer any issues. Users likely saw a brief spike in 503s, and then elevated TTFAs (backup cluster) for a few minutes, entire incident would have resolved in a few minutes. Will have an RCA shortly
- postmortem Dec 18, 2025, 08:10 PM UTC
We had a faulty deploy to the US cluster that would have resulted in a brief spike in 503s. Newer traffic was rerouted to our backup cluster \(EU\) while rollback initiated in the US which took a minute\*.
Critical November 14, 2025 CRUD endpoint outage 33m Playground
Started Nov 14, 2025, 02:29 AM UTC· resolved Nov 14, 2025, 03:03 AM UTC
- investigating Nov 14, 2025, 02:29 AM UTC
The Cartesia playground is currently down. We are investigating.
- resolved Nov 14, 2025, 03:03 AM UTC
The playground should be back up. The playground outage was downstream of a more general outage on CRUD endpoints caused by a database issue which is now resolved. Thanks to previous infra hardening efforts, this issue did not affect production traffic on our real-time APIs, including TTS & STT.
Notice November 10, 2025 EU Specific Route 404 11m Text to Speech (EU)
Started Nov 10, 2025, 07:53 AM UTC· resolved Nov 10, 2025, 08:04 AM UTC
- identified Nov 10, 2025, 07:53 AM UTC
As part of a scheduled node migration, the EU specific domain was routed to a new node. This appears to not be working as intended, we're deploying a fix shortly.
- monitoring Nov 10, 2025, 07:55 AM UTC
We've deployed a fix for the route.
- monitoring Nov 10, 2025, 07:58 AM UTC
We are continuing to monitor for any further issues.
- resolved Nov 10, 2025, 08:04 AM UTC
Users appear to confirm that this has been resolved since. Will update retroactively with a more thorough RCA.
- postmortem Nov 10, 2025, 08:34 AM UTC
We had a scheduled node migration during this timeframe that was intended to be a no-op. The intermediary cluster to which the `api-eu` domain specifically was routed to was not expected to affect, but unfortunately the `api-eu` was not properly whitelisted. This was a gap in our health checks that we did not catch this immediately and rollback, we'll be updating that in our pipeline. The primary `api` route with geosteering was unaffected, even if called from the EU domain.
Minor November 6, 2025 Database connection issue leading to impacted Agents and Narrations 22m Playground
Started Nov 06, 2025, 07:16 PM UTC· resolved Nov 06, 2025, 07:38 PM UTC
- investigating Nov 06, 2025, 07:16 PM UTC
We are experiencing some database connection issues impacting some product service area including Agents and Narrations. These do not impact TTS, Voice or Cloning APIs.
- monitoring Nov 06, 2025, 07:38 PM UTC
Root cause was determined to be an unforeseen networking configuration change from an upstream provider. Mitigations have been applied, all errors are resolved, and we’re closely monitoring the system.
- resolved Nov 06, 2025, 07:38 PM UTC
This incident has been resolved.
Minor November 4, 2025 Pro Voice Cloning: Long training times 1d Voice Cloning
Started Nov 04, 2025, 07:52 PM UTC· resolved Nov 05, 2025, 08:49 PM UTC
- investigating Nov 04, 2025, 07:52 PM UTC
Pro Voice Cloning is experiencing some hangs in the training pipeline. We are investigating into the root cause of PVCs being stuck and not reaching completion or failure.
- investigating Nov 04, 2025, 09:44 PM UTC
We found a serialization issue that is causing uncaught errors in our training pipeline. We are working on a fix for both the serialization bug and improved error handling to communicate failed training pipelines to users.
- identified Nov 05, 2025, 02:49 AM UTC
We have identified the root cause. The issue is impacting only a small number of requests and we are providing support for users. The full resolution will be resolved in the next Morning to ensure stability of our other services.
- resolved Nov 05, 2025, 08:49 PM UTC
This incident has been resolved.
Major October 14, 2025 Elevated errors on TTS & STT API 2h 29m Text to Speech (US), Speech to Text (US)
Started Oct 14, 2025, 08:47 PM UTC· resolved Oct 14, 2025, 11:17 PM UTC
- investigating Oct 14, 2025, 08:47 PM UTC
We are currently investigating this issue.
- investigating Oct 14, 2025, 08:47 PM UTC
We are continuing to investigate this issue.
- identified Oct 14, 2025, 08:52 PM UTC
We're facing an outage with an infrastructure provider, we're actively working on a fallback.
- identified Oct 14, 2025, 09:34 PM UTC
We've re-routed traffic to our other clusters; the API is still degraded due to elevated latencies on TTS & STT services.
- identified Oct 14, 2025, 10:32 PM UTC
We are still working with the primary provider to bring the cluster back up, but in the meanwhile traffic is served from our global clusters at about `100-200ms` higher latencies for our US traffic.
- identified Oct 14, 2025, 11:16 PM UTC
We've rolled out a fix and US traffic is being routed to our US clusters. Latencies should be within normal ranges now.
- resolved Oct 15, 2025, 12:35 AM UTC
This incident has now been resolved, we'll follow up with an RCA shortly!
- postmortem Oct 15, 2025, 11:45 PM UTC
## Incident summary On October 14, Cartesia’s API experienced a major outage between **1:38 PM and 4:16 PM PT**, resulting in **10–15 minutes of complete downtime**, **45 minutes of high latency**, and **1 hour 45 minutes of degraded performance** with elevated response times of **\+200–400 ms**. The root cause was that the Kubernetes control, network, and compute planes shared nodes, allowing unbounded workloads from a new deployment to overload control plane nodes and trigger cascading failures. The issue was resolved after isolating and remediating the affected workloads. To prevent recurrence, we are enforcing pod resource limits, isolating control/ingress/compute planes, and expanding regional surge capacity to improve recovery times in future incidents. ## Impact * 10-15 minutes of complete outage * 45 minutes of high latencies while bringing up regional clusters to serviceable capacity * 1 hour 45 minutes of overall elevated latencies serving from EU/APAC ## Root Cause Analysis The core contributor to this issue was that in our primary inference cluster, the Kubernetes control plane, network plane, and compute plane didn’t have node level isolation. As a result, the `K8s` API Server, Ingress controllers and custom workloads could be colocated on the same node. At `1:38 PM`, we deployed workloads with a high replica count. These workloads were scheduled on the controller plane nodes, which overloaded the nodes. During the process of recovery on the controller nodes, the workloads were rescheduled on the same nodes, causing a thundering herd of crash loops. The root cause was identified after about `20 minutes`, but full restoration was delayed as the underlying nodes required manual intervention from our cloud service provider \(CSP\). By `4:12 PM`, the cluster admins were able to cordon the Control Plane, bring it back, and delete the bad workloads, resuming service to our remaining cluster. ## Detection and Timeline * `1:38 PM` - Got pages for primary API cluster being inaccessible. * `1:40 PM` - Pinged our primary CSP that the cluster is inaccessible via SSH * `1:50 PM` - Sharded traffic globally across different AZs * `2:20 PM` - Added sufficient surge capacity across different AZs to serve traffic stably. * `3:49 PM` - CSP reports cluster is back up. However, the Ingress is still unusable due to the head nodes being cordoned. * `4:16 PM` - Ingress and Control plane are back up, traffic is moved back to US and stable ## Lessons learned and Next Steps * `Prevention` : Running a workload with unbounded resource limits should not be permitted. We ran a safe workload for a typical compute node, but in the case of a much smaller control plane node, the lack of limits OOMed the entire node. * Moving forward, we’re adding validations to ensure that all workloads have a `requests` and `limits` field configured for our `Pods`. * `Prevention` : A shared `K8s` control, ingress and compute plane is not common practice. We are working with the cloud service provider to get this remediated and move to more standard practices around complete separation of concerns. * The aimed solution here is to have an `Unschedulable` or Cordoned set of Control Plane nodes \(In the case of CSPs like AWS, these are usually not even exposed\), and a separate set of nodes for managing Ingress controllers. * `Mitigation` : We are working on increasing our provisioned capacity on backup clusters on different CSP in the same locale. This will increase robustness to failures in a single CSP and ensure fast failover recovery.