Cartesia Outage History

## Overview On March 4, 2026, our platform experienced a period of degraded performance and a brief service interruption, primarily impacting users in the APAC region. The incident was triggered by an update to our Load Balancing layer that inadvertently extended to the cache synchronization workers across several global regions. This synchronization initiated a high-volume data retrieval process from our primary database. The resulting resource contention coincided with peak organic traffic, leading to increased latency and, eventually, a necessary database restart to restore system stability. ## Incident Timeline \(PST\) - March 4th, 2026 * **15:50 – 16:50:** A new version deployment triggered a phased restart of cache synchronization workers across global regions. * **18:30:** Coinciding with peak regional demand, the primary database experienced elevated CPU utilization. Users began noticing increased Time to First Audio \(TTFA\). * **19:45:** Database resources reached a critical threshold, leading to broader performance degradation. * **20:18:** Engineering teams initiated an investigation into database health. * **21:00:** To clear queued transactions and stabilize the environment, a manual reset of the primary database was performed, resulting in a brief total service interruption. * **21:04:** Database services were successfully restored. * **21:08:** Latency metrics returned to normal levels. * **21:12:** All background cache synchronization tasks successfully completed. ## Root Cause Analysis ### Resource Contention and Buffer Saturation The primary driver of this incident was the simultaneous "cold start" of cache workers across multiple regions. By design, these workers attempt to populate the rapid-access cache by performing a comprehensive data fetch from the primary database. Because several regions initiated this process concurrently, the database was subjected to multiple long-running read operations. This created a high demand for the database’s buffer pool. When peak organic traffic arrived, the database struggled to balance these synchronization read requests with the high-frequency write operations required by active user sessions. ### Latent Dependency A recent configuration update introduced an unexpected synchronous database look-up during the request resolution path. While typically negligible, this added DB query time became significant when the DB was under heavy load. This dependency meant that as database performance slowed, user-facing latency increased proportionally, eventually leading to a backlog of requests. ## Lessons Learned and Resolution ### Architectural Optimization * **Decoupled Synchronization:** We are re-evaluating our cache synchronization process to make it future proof as our load grows. * **Improving our global database architecture:** We are implementing a global database architectural change to make our system more resilient. ### Enhanced Monitoring We are deploying a unified health dashboard to provide real-time visibility into: * Granular resolution request hot path latency. * More aggressive alerting on DB / Cache health.

# Overview * During a maintenance upgrade to Cartesia’s voice metadata, existing metadata for a narrow subset of default Cartesia voices was unintentionally overwritten, causing voice changes. * Some voices were affected more than others. A narrow subsection of voices saw increased hallucinations while 3 voices were significantly impacted. * Degradation occurred gradually as caches began expiring and was mitigated after a database restore and global cache purge. * In addition to fixing the root cause, we are updating our change management and monitoring process to prevent issues like this in the future by enforcing data upgrade safety via tooling, improving automatic detection, updating our triage process, and investing in a lower recovery time objective \(RTO\) for critical data like voices. # Detailed Analysis ## Timeline \(UTC\) * **2026-02-25 04:57** – Upgrade completes, overwritten voice metadata starts being served as caches expire. We regularly test our model for regression, but since only a small subsection of voices were affected, they were not caught by our automated testing in this case. * **2026-02-25 23:00 - 2026-02-26 17:00** – We receive reports of voice changes and begin internal investigation to assess scope and severity. The on-call investigated but made an incorrect determination of the source and severity of the issue which delayed remediation. * **2026-02-26 18:40** – Impact triaged to high severity. * **2026-02-26 19:27** – Root cause identified; mitigation plan begins. * **2026-02-26 22:13** – Data fully restored; global cache purge begins. * **2026-02-26 23:06** – Global cache purge completes. Customers begin confirming resolution. * **2026-02-26 23:20** – Status page marked resolved ## Root cause A bug in the code executing a maintenance upgrade to Cartesia’s voice metadata caused the metadata for some existing voices to be regenerated and overwritten. This metadata is fundamental to how we represent voices and changes to this metadata can lead to changes in voice output. The code path containing the bug was executed because some default voices did not fulfill an invariant of the metadata upgrade process and were incorrectly identified as requiring an upgrade. The issue was not caught in manual or automated testing, because because it only reproduces in a specific state that exists in production and affects a narrow subsection of voices. ## Learnings and Next Steps We sincerely apologize for this incident and the disruption it caused to your business. We understand that reliable voice quality is fundamental to your trust in Cartesia. We are making the following corrections to our change management process and monitoring process to prevent issues like this in the future, targeting every step of the release and error recovery lifecycle: 1. **Safer processes for routine data changes, enforced via tooling:** We are investing in automated tooling to make our existing change management process even more thorough. We will incorporate additional automated tooling to triage the risk of changes, do dry runs on realistic data, and ensure multiple sign-offs beyond code review with automatic rollbacks in case of errors. 2. **Improved automatic detection**: We are expanding our automatic voice regression testing to expand coverage to much larger set of voices and transcripts. 3. **Update triage process:** The issue was incorrectly triaged. We will update our triage playbook to ensure that any reported voice issues are comprehensively checked against recent code and data changes to speed up an RCA and determine scope. This will ensure future issues are correctly escalated faster. 4. **Invest in lower RTO:** Our RTO for our voices infra is currently at 4 hours. However, for critical data like voices, we will invest in bringing down our RTO to 15 minutes. We will implement automation around finer grained data snapshots and data restoration that are currently manual or slow. If you have any questions or concerns about this incident, please don't hesitate to reach out to our support team.