Cartesia Outage History

Started Jun 16, 2026, 11:57 PM UTC· resolved Jun 17, 2026, 04:00 AM UTC

1. identified Jun 16, 2026, 11:57 PM UTC

   We are experiencing infrastructure issue with one of our infra provider, mainly impacting our voice clone feature in the US.

2. identified Jun 16, 2026, 11:57 PM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Jun 17, 2026, 12:37 AM UTC

   We have mitigated the issue on our end and are continuing to monitor the state of our service.

4. resolved Jun 17, 2026, 04:28 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 16, 2026, 09:30 PM UTC· resolved Jun 16, 2026, 09:30 PM UTC

1. resolved Jun 16, 2026, 10:40 PM UTC

   There was a partial outage for ink-whisper due to which traffic being routed to a particular pod worker was failing. We have rolled out a fix for this to prevent this from happening again.

Read the full incident report →

Started Jun 16, 2026, 05:11 PM UTC· resolved Jun 16, 2026, 05:33 PM UTC

1. investigating Jun 16, 2026, 05:11 PM UTC

   We are currently investigating the issue.

2. monitoring Jun 16, 2026, 05:25 PM UTC

   A fix has been implemented to address agent deployment and call connection errors. We are monitoring the results.

3. resolved Jun 16, 2026, 05:33 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 11, 2026, 05:12 PM UTC· resolved Jun 11, 2026, 06:54 PM UTC

1. investigating Jun 11, 2026, 05:12 PM UTC

   We are aware of elevated training durations for Pro Voice Clones, the team is actively investigating.

2. monitoring Jun 11, 2026, 05:45 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 11, 2026, 06:54 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 09, 2026, 03:06 PM UTC· resolved Jun 09, 2026, 03:24 PM UTC

1. investigating Jun 09, 2026, 03:05 PM UTC

   We're experiencing an elevated level of errors downloading files and are currently looking into the issue.

2. identified Jun 09, 2026, 03:06 PM UTC

   Our file server is unable to verify read permissions. Most requests should succeed now.

3. identified Jun 09, 2026, 03:10 PM UTC

   We are continuing to work on a fix for this issue.

4. resolved Jun 09, 2026, 03:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 09, 2026, 01:00 PM UTC· resolved Jun 09, 2026, 01:00 PM UTC

1. resolved Jun 09, 2026, 02:50 PM UTC

   We're experiencing errors downloading files.

2. postmortem Jun 09, 2026, 02:51 PM UTC

   Our file server could not verify read permissions.

Read the full incident report →

Started Jun 01, 2026, 07:14 PM UTC· resolved Jun 01, 2026, 08:15 PM UTC

1. investigating Jun 01, 2026, 07:14 PM UTC

   We are currently investigating timeouts on API requests to CRUD endpoints (including Voices & Agents).

2. investigating Jun 01, 2026, 07:26 PM UTC

   We are continuing to investigate - users accessing the Playground may also be affected by this.

3. monitoring Jun 01, 2026, 07:54 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jun 01, 2026, 10:26 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 01, 2026, 04:58 PM UTC· resolved Jun 01, 2026, 05:10 PM UTC

1. investigating Jun 01, 2026, 04:58 PM UTC

   Due to issues with a downstream provider, calls made to voice agents on the Line platform may fail to connect

2. investigating Jun 01, 2026, 05:10 PM UTC

   We are continuing to investigate this issue.

3. resolved Jun 01, 2026, 05:10 PM UTC

   The issue with our downstream provider has been resolved.

Read the full incident report →

Started May 26, 2026, 09:36 PM UTC· resolved May 26, 2026, 10:00 PM UTC

1. identified May 26, 2026, 09:36 PM UTC

   We have identified an issue with our authentication provider that is currently affecting the Cartesia playground. No production API traffic is affected.

2. resolved May 27, 2026, 01:04 AM UTC

   We have confirmed resolution of the incident with our auth provider.

Read the full incident report →

Started May 26, 2026, 02:00 PM UTC· resolved May 26, 2026, 02:00 PM UTC

1. resolved May 26, 2026, 03:32 PM UTC

   Our TTS endpoints for sonic-3.5 experienced degraded performance and queuing between UTC 15:01 and UTC 15:32. We've mitigated the issue and are monitoring the situation.

Read the full incident report →

Started May 20, 2026, 03:47 PM UTC· resolved May 20, 2026, 03:47 PM UTC

1. monitoring May 20, 2026, 03:47 PM UTC

   Due to some billing issues with our telephony provider, phone calls were unavailable from 7:54 AM PST to 8:39 AM PST. We've since resolved the issue and are looking into how to improve our processes further to avoid issues like this in the future. Please let us know if you continue to see any errors, we apologize for the inconvenience.

2. resolved May 20, 2026, 03:47 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 12, 2026, 09:28 PM UTC· resolved May 13, 2026, 03:35 AM UTC

1. investigating May 12, 2026, 09:28 PM UTC

   We are noticing issues with some services that require a path to blob stores: The following are impacted: - Voice agent call recordings (creating and downloading) - Voice Cloning Creation - Pro Voice Cloning The following are *NOT impacted*: - Our TTS API - The voice agent call will complete (just no call recordings)

2. investigating May 12, 2026, 09:56 PM UTC

   We are continuing to investigate this issue.

3. investigating May 12, 2026, 11:26 PM UTC

   We deployed some mitigations and most services are restored. The only degradation we face is that voice agent call recordings are currently delayed.

4. monitoring May 12, 2026, 11:29 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 13, 2026, 03:35 AM UTC

   This incident has been resolved. All voice agent call recordings should be visible on the playground. Please let us know if you continue to experience any issues.

Read the full incident report →

Started May 08, 2026, 01:24 AM UTC· resolved May 08, 2026, 03:00 AM UTC

1. investigating May 08, 2026, 01:24 AM UTC

   We are currently investigating this issue.

2. investigating May 08, 2026, 01:24 AM UTC

   The issue seems to be stemming from an upstream provider.

3. monitoring May 08, 2026, 02:52 AM UTC

   A mitigation has been rolled out and we're monitoring for recovery.

4. resolved May 08, 2026, 03:00 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 05, 2026, 09:48 PM UTC· resolved May 05, 2026, 10:10 PM UTC

1. investigating May 05, 2026, 09:48 PM UTC

   We are currently investigating an issue with playground pages taking a long time to load. Initial investigation indicates the issue started at 2:37 PM PT (10m ago).

2. investigating May 05, 2026, 09:56 PM UTC

   We have identified and mitigated the issue. We are monitoring the fix.

3. resolved May 05, 2026, 10:10 PM UTC

   We have confirmed the fix had its intended effect. This incident is resolved.

Read the full incident report →

Started Apr 28, 2026, 01:41 AM UTC· resolved Apr 28, 2026, 02:20 AM UTC

1. investigating Apr 28, 2026, 01:41 AM UTC

   We are investigating an issue causing some voice agents are failing to reach their runtime deployments, returning a "trouble connecting to an agent" error. We will share an update as soon as we have more information.

2. identified Apr 28, 2026, 02:13 AM UTC

   We've identified an issue with an upstream provider, and we're actively working on rolling it back.

3. resolved Apr 28, 2026, 02:20 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 02, 2026, 08:07 PM UTC· resolved Apr 02, 2026, 09:28 PM UTC

1. investigating Apr 02, 2026, 08:07 PM UTC

   We are currently investigating this issue.

2. monitoring Apr 02, 2026, 09:22 PM UTC

   An upstream infrastructure provider identified a DNS resolution issue affecting WebSocket connections to voice agent deployments. A mitigation has been put in place and failure rates have recovered fully. We are continuing to monitor.

3. resolved Apr 02, 2026, 09:28 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 01, 2026, 04:00 PM UTC· resolved Apr 01, 2026, 04:00 PM UTC

1. resolved Apr 01, 2026, 05:04 PM UTC

   Our application was deployed on a node that had falsely configured networking setup, resulted in degradation of our TTS service in the US. Customers may see elevated connection errors and false rate limiting errors. The incident happened between 9:04AM and 9:08AM PT.

Read the full incident report →

Started Mar 13, 2026, 02:02 PM UTC· resolved Mar 13, 2026, 08:45 PM UTC

1. investigating Mar 13, 2026, 11:41 PM UTC

   We are experiencing intermittent connectivity issues affecting voice agent sessions due to degradation at an upstream infrastructure provider. Some calls may fail to connect or drop unexpectedly.

2. identified Mar 13, 2026, 11:41 PM UTC

   A mitigation and additional observability have been deployed. We are actively monitoring for recovery and are seeing connections recover.

3. resolved Mar 13, 2026, 11:42 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 10, 2026, 04:45 PM UTC· resolved Mar 10, 2026, 05:49 PM UTC

1. monitoring Mar 10, 2026, 04:45 PM UTC

   We're currently experiencing degraded login functionality on play.cartesia.ai due to an issue with our authentication provider, Clerk. We're monitoring the situation and you can also track their status at: https://status.clerk.com/

2. resolved Mar 10, 2026, 05:49 PM UTC

   This incident has been resolved by Clerk.

Read the full incident report →

Started Mar 10, 2026, 04:00 PM UTC· resolved Mar 10, 2026, 04:00 PM UTC

1. resolved Mar 11, 2026, 05:26 AM UTC

   Duration: March 10, 2026, 9:00 AM PST - 9:14 PM PST Impact: Runtime logs for Line agent calls made during this time period were not available via the dashboard or API. Call processing was not affected. Resolution: Our engineering team identified the root cause as an upstream breaking change from an infrastructure provider that disrupted our log ingestion pipeline. A hotfix was deployed and log collection has been fully restored. All logs from the affected window have also now been recovered and backfilled. Next Steps: We are conducting a postmortem to improve reliability and detection speed moving forward.

Read the full incident report →

Started Mar 05, 2026, 05:40 AM UTC· resolved Mar 05, 2026, 06:01 AM UTC

1. investigating Mar 05, 2026, 04:55 AM UTC

   We are currently investigating elevated Time to First Audio (TTFA) for api.cartesia.ai in the APAC region. Customers may be experiencing longer-than-expected delays before audio begins streaming.

2. identified Mar 05, 2026, 05:40 AM UTC

   We have identified the issue and patched a possible fix. We are continuing to monitor the issue.

3. resolved Mar 05, 2026, 06:01 AM UTC

   The issue has been resolved

4. postmortem Mar 10, 2026, 11:56 PM UTC

   ## Overview On March 4, 2026, our platform experienced a period of degraded performance and a brief service interruption, primarily impacting users in the APAC region. The incident was triggered by an update to our Load Balancing layer that inadvertently extended to the cache synchronization workers across several global regions. This synchronization initiated a high-volume data retrieval process from our primary database. The resulting resource contention coincided with peak organic traffic, leading to increased latency and, eventually, a necessary database restart to restore system stability. ## Incident Timeline \(PST\) - March 4th, 2026 * **15:50 – 16:50:** A new version deployment triggered a phased restart of cache synchronization workers across global regions. * **18:30:** Coinciding with peak regional demand, the primary database experienced elevated CPU utilization. Users began noticing increased Time to First Audio \(TTFA\). * **19:45:** Database resources reached a critical threshold, leading to broader performance degradation. * **20:18:** Engineering teams initiated an investigation into database health. * **21:00:** To clear queued transactions and stabilize the environment, a manual reset of the primary database was performed, resulting in a brief total service interruption. * **21:04:** Database services were successfully restored. * **21:08:** Latency metrics returned to normal levels. * **21:12:** All background cache synchronization tasks successfully completed. ## Root Cause Analysis ### Resource Contention and Buffer Saturation The primary driver of this incident was the simultaneous "cold start" of cache workers across multiple regions. By design, these workers attempt to populate the rapid-access cache by performing a comprehensive data fetch from the primary database. Because several regions initiated this process concurrently, the database was subjected to multiple long-running read operations. This created a high demand for the database’s buffer pool. When peak organic traffic arrived, the database struggled to balance these synchronization read requests with the high-frequency write operations required by active user sessions. ### Latent Dependency A recent configuration update introduced an unexpected synchronous database look-up during the request resolution path. While typically negligible, this added DB query time became significant when the DB was under heavy load. This dependency meant that as database performance slowed, user-facing latency increased proportionally, eventually leading to a backlog of requests. ## Lessons Learned and Resolution ### Architectural Optimization * **Decoupled Synchronization:** We are re-evaluating our cache synchronization process to make it future proof as our load grows. * **Improving our global database architecture:** We are implementing a global database architectural change to make our system more resilient. ### Enhanced Monitoring We are deploying a unified health dashboard to provide real-time visibility into: * Granular resolution request hot path latency. * More aggressive alerting on DB / Cache health.

Read the full incident report →

Started Feb 26, 2026, 08:45 PM UTC· resolved Feb 26, 2026, 11:20 PM UTC

1. identified Feb 26, 2026, 09:06 PM UTC

   We have identified a degradation in speech generation quality for requests using public voices since 6:30 PM PST on February 25.

2. identified Feb 26, 2026, 09:08 PM UTC

   We are continuing to work on a fix for this issue.

3. identified Feb 26, 2026, 10:14 PM UTC

   We are continuing to work on a fix for this issue. A resolution is expected in 1 hour.

4. monitoring Feb 26, 2026, 10:44 PM UTC

   We have rolled out a fix globally that should slowly take place over the next 10-15 minutes.

5. monitoring Feb 26, 2026, 11:11 PM UTC

   We confirmed that the fix had been rolled out and TTS generation quality should resume to normal. We are continuing to monitor the situation to ensure that the incident had been resolved.

6. resolved Feb 26, 2026, 11:20 PM UTC

   The incident has been resolved

7. postmortem Feb 28, 2026, 02:18 AM UTC

   # Overview * During a maintenance upgrade to Cartesia’s voice metadata, existing metadata for a narrow subset of default Cartesia voices was unintentionally overwritten, causing voice changes. * Some voices were affected more than others. A narrow subsection of voices saw increased hallucinations while 3 voices were significantly impacted. * Degradation occurred gradually as caches began expiring and was mitigated after a database restore and global cache purge. * In addition to fixing the root cause, we are updating our change management and monitoring process to prevent issues like this in the future by enforcing data upgrade safety via tooling, improving automatic detection, updating our triage process, and investing in a lower recovery time objective \(RTO\) for critical data like voices. # Detailed Analysis ## Timeline \(UTC\) * **2026-02-25 04:57** – Upgrade completes, overwritten voice metadata starts being served as caches expire. We regularly test our model for regression, but since only a small subsection of voices were affected, they were not caught by our automated testing in this case. * **2026-02-25 23:00 - 2026-02-26 17:00** – We receive reports of voice changes and begin internal investigation to assess scope and severity. The on-call investigated but made an incorrect determination of the source and severity of the issue which delayed remediation. * **2026-02-26 18:40** – Impact triaged to high severity. * **2026-02-26 19:27** – Root cause identified; mitigation plan begins. * **2026-02-26 22:13** – Data fully restored; global cache purge begins. * **2026-02-26 23:06** – Global cache purge completes. Customers begin confirming resolution. * **2026-02-26 23:20** – Status page marked resolved ## Root cause A bug in the code executing a maintenance upgrade to Cartesia’s voice metadata caused the metadata for some existing voices to be regenerated and overwritten. This metadata is fundamental to how we represent voices and changes to this metadata can lead to changes in voice output. The code path containing the bug was executed because some default voices did not fulfill an invariant of the metadata upgrade process and were incorrectly identified as requiring an upgrade. The issue was not caught in manual or automated testing, because because it only reproduces in a specific state that exists in production and affects a narrow subsection of voices. ## Learnings and Next Steps We sincerely apologize for this incident and the disruption it caused to your business. We understand that reliable voice quality is fundamental to your trust in Cartesia. We are making the following corrections to our change management process and monitoring process to prevent issues like this in the future, targeting every step of the release and error recovery lifecycle: 1. **Safer processes for routine data changes, enforced via tooling:** We are investing in automated tooling to make our existing change management process even more thorough. We will incorporate additional automated tooling to triage the risk of changes, do dry runs on realistic data, and ensure multiple sign-offs beyond code review with automatic rollbacks in case of errors. 2. **Improved automatic detection**: We are expanding our automatic voice regression testing to expand coverage to much larger set of voices and transcripts. 3. **Update triage process:** The issue was incorrectly triaged. We will update our triage playbook to ensure that any reported voice issues are comprehensively checked against recent code and data changes to speed up an RCA and determine scope. This will ensure future issues are correctly escalated faster. 4. **Invest in lower RTO:** Our RTO for our voices infra is currently at 4 hours. However, for critical data like voices, we will invest in bringing down our RTO to 15 minutes. We will implement automation around finer grained data snapshots and data restoration that are currently manual or slow. If you have any questions or concerns about this incident, please don't hesitate to reach out to our support team.

Read the full incident report →

Started Feb 19, 2026, 06:21 PM UTC· resolved Feb 19, 2026, 06:45 PM UTC

1. investigating Feb 19, 2026, 06:21 PM UTC

   TTS generations using pronunciation dictionaries produce incorrect audio (some structured metadata is read out in the speech). Workaround for affected customers: use `sonic-3` instead of `sonic-3-latest`.

2. investigating Feb 19, 2026, 06:45 PM UTC

   We have rolled back the bad deploy and pronunciation dictionaries are once again working with `sonic-3-latest`.

3. resolved Feb 19, 2026, 06:45 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 19, 2026, 05:12 PM UTC· resolved Feb 19, 2026, 06:41 PM UTC

1. investigating Feb 19, 2026, 05:12 PM UTC

   Some users are having trouble signing in; some users are experiencing latency in signin. Our provider Clerk has an incident page here: https://status.clerk.com/ API and API authorization is unaffected; this is only on play.cartesia.ai and docs.cartesia.ai

2. monitoring Feb 19, 2026, 06:00 PM UTC

   Our provider has rolled out a fix and we are monitoring the recovery.

3. resolved Feb 19, 2026, 06:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 09, 2026, 05:05 PM UTC· resolved Feb 09, 2026, 06:00 PM UTC

1. investigating Feb 09, 2026, 05:16 PM UTC

   We've noticed errors when generating audio on `/bytes` and `/sse` endpoints in India. We are currently investigating this issue.

2. monitoring Feb 09, 2026, 05:55 PM UTC

   We've identified and mitigated a DDoS attack on our APAC infrastructure. Traffic protection measures are now in place and routing has been restored. We're continuing to monitor the situation to ensure stability. Error rates have been normalized.

3. resolved Feb 09, 2026, 06:00 PM UTC

   This incident has been resolved.

Read the full incident report →