TTS API Elevated TTFA

Update timeline

1. investigating Mar 05, 2026, 04:55 AM UTC

   We are currently investigating elevated Time to First Audio (TTFA) for api.cartesia.ai in the APAC region. Customers may be experiencing longer-than-expected delays before audio begins streaming.

2. identified Mar 05, 2026, 05:40 AM UTC

   We have identified the issue and patched a possible fix. We are continuing to monitor the issue.

3. resolved Mar 05, 2026, 06:01 AM UTC

   The issue has been resolved

4. postmortem Mar 10, 2026, 11:56 PM UTC

   ## Overview On March 4, 2026, our platform experienced a period of degraded performance and a brief service interruption, primarily impacting users in the APAC region. The incident was triggered by an update to our Load Balancing layer that inadvertently extended to the cache synchronization workers across several global regions. This synchronization initiated a high-volume data retrieval process from our primary database. The resulting resource contention coincided with peak organic traffic, leading to increased latency and, eventually, a necessary database restart to restore system stability. ## Incident Timeline \(PST\) - March 4th, 2026 * **15:50 – 16:50:** A new version deployment triggered a phased restart of cache synchronization workers across global regions. * **18:30:** Coinciding with peak regional demand, the primary database experienced elevated CPU utilization. Users began noticing increased Time to First Audio \(TTFA\). * **19:45:** Database resources reached a critical threshold, leading to broader performance degradation. * **20:18:** Engineering teams initiated an investigation into database health. * **21:00:** To clear queued transactions and stabilize the environment, a manual reset of the primary database was performed, resulting in a brief total service interruption. * **21:04:** Database services were successfully restored. * **21:08:** Latency metrics returned to normal levels. * **21:12:** All background cache synchronization tasks successfully completed. ## Root Cause Analysis ### Resource Contention and Buffer Saturation The primary driver of this incident was the simultaneous "cold start" of cache workers across multiple regions. By design, these workers attempt to populate the rapid-access cache by performing a comprehensive data fetch from the primary database. Because several regions initiated this process concurrently, the database was subjected to multiple long-running read operations. This created a high demand for the database’s buffer pool. When peak organic traffic arrived, the database struggled to balance these synchronization read requests with the high-frequency write operations required by active user sessions. ### Latent Dependency A recent configuration update introduced an unexpected synchronous database look-up during the request resolution path. While typically negligible, this added DB query time became significant when the DB was under heavy load. This dependency meant that as database performance slowed, user-facing latency increased proportionally, eventually leading to a backlog of requests. ## Lessons Learned and Resolution ### Architectural Optimization * **Decoupled Synchronization:** We are re-evaluating our cache synchronization process to make it future proof as our load grows. * **Improving our global database architecture:** We are implementing a global database architectural change to make our system more resilient. ### Enhanced Monitoring We are deploying a unified health dashboard to provide real-time visibility into: * Granular resolution request hot path latency. * More aggressive alerting on DB / Cache health.