LiveKit Outage History

## Summary LiveKit's core realtime and agent services are designed to tolerate database failures. WebRTC media, SIP calls, and hosted agent sessions continue to operate even when our database backend is slow or unavailable. A subset of Room APIs, specifically `CreateRoom`, `DeleteRoom`, and `UpdateRoomMetadata`, do depend on a database for consistency and disaster recovery. That database is highly available and globally distributed, with no single point of failure. When it is under significant contention, these Room APIs can return errors or time out, while realtime traffic continues to flow normally. On 2026-04-15, database contention caused a percentage of Room API calls to fail in our US-West region. Remediation work later produced a 26-minute global outage of the Room APIs. Realtime sessions, SIP calls, and agent processes were unaffected throughout. We sincerely apologize to customers whose applications were disrupted during this incident. ## Impact The incident had two distinct phases of customer impact. ### Phase 1: Elevated Room API timeouts in US-West \(2026-04-15 22:10 UTC to 2026-04-16 03:14 UTC\) A percentage of `DeleteRoom`, `UpdateRoomMetadata`, and `ListRooms` calls timed out, primarily in our US-West region. Other regions saw limited impact during this phase. Customers with high Room API volume in US-West observed elevated error rates on their integrations; the majority of customers were not affected. ### Phase 2: Global Room API outage \(2026-04-16 03:14 to 03:40 UTC, ~26 minutes\) While we were swapping in a rebuilt `rooms` table, the table was briefly missing from the database, and the majority of Room API calls globally returned HTTP 500 with `ERROR: relation "rooms" does not exist`. WebRTC sessions, SIP calls, and agent processes continued to function, and realtime connection counts remained stable. Applications that depend on Room APIs to start or manage sessions saw visible failures during this window. ## Root Cause The sweeper is a background process that removes rows from the `rooms` table as sessions end. Earlier on 2026-04-15, its throughput dropped significantly, and over roughly 8 hours stale rows accumulated to the point where the table was many times larger than its intended steady-state size. At approximately 21:00 UTC, a routine schema migration was applied to a different, unrelated table. The migration itself did not touch `rooms`, but it raised overall database disk utilization and background load. Combined with the oversized `rooms` table, this produced enough contention to slow down reads and writes against it. The effect first appeared in US-West, where the regional mix of Room API traffic was most sensitive to the contention. Once we identified the oversized table as the underlying cause, we needed to restore it to a healthy size. Because the table was already contended, deleting rows directly would have taken additional locks and worsened the contention. We instead chose to rebuild the table: create a new table with the same schema, copy over the active rows, then atomically swap the new table into place via a pair of renames. The copy phase completed quickly. The first rename \(moving the old `rooms` table aside\) completed in about 2.5 minutes. The second rename, moving the new table into the `rooms` name, stalled on our globally distributed database for significantly longer than we anticipated. During the stall, the `rooms` table did not exist from the perspective of any region, and all Room API calls globally returned errors. After roughly 10 minutes, we aborted the stalled rename, created a fresh `rooms` table from scratch, and inserted the active rows into it. Room API traffic recovered globally shortly after. ## Corrective Actions & Prevention The following improvements have been implemented or initiated to reduce the likelihood and impact of similar incidents: * **Enhance monitoring for sweeper throughput and active room count.** We are adding and hardening alerts on sweeper throughput and active room count, so that any future divergence pages on-call well before it threatens production. * **Improve sweeper resilience and throughput.** We are investigating the cause of the sweeper's throughput drop and adding capacity headroom so a transient slowdown cannot translate into multi-hour backlog growth. * **Remove database as a dependency for Room APIs**. This incident reaffirmed our long-held design principle that realtime services should not depend on databases. We believe this is the only way to build a system that approaches 100% uptime, and we will continue the work to ensure Room APIs do not depend on a database either. The Phase 2 outage was caused by our own remediation, and we recognize how disruptive it was for applications that depend on the Room APIs. We are committed to the work above to reduce both the likelihood and the blast radius of a similar failure in the future. Thank you for your patience, and we welcome any additional feedback from customers who were affected.

**Root Cause** A gap in our DNS update process resulted in a misconfiguration of the DNS for [livekit.io](http://livekit.io), causing several A records to be overwritten. This resulted in an incorrect IP address being returned for [livekit.io](http://livekit.io). All real-time services - calls, RTC, and hosted agents - were unaffected as they operate on separate domains. We do have DNS monitoring in place, but it was not configured to page on-call. As a result, the issue was identified by LiveKit engineering after a short delay. **Timeline** * **2026-04-03 17:46 UTC**: DNS configuration change applied; resolution begins failing for [livekit.io](http://livekit.io). * **2026-04-03 17:52 UTC**: Issue identified by LiveKit. * **2026-04-03 18:15 UTC**: Root cause identified and fix applied. * **2026-04-03 18:18 UTC**: Fix fully propagated; services restored. Total duration: ~32 minutes. **Mitigations** * Enforce tighter automated guardrails around DNS updates. * Enable paging on existing DNS health checks so any future issues are caught even sooner.

**Root Cause** During a failover event on the SIP load balancer in US East, UDP packets were incorrectly treated as part of an existing "connection" by the underlying VNIC stack's connection tracking \(conntrack\) mechanism. As a result, these packets continued to be forwarded to the previous \(now non-existent\) node, causing SIP INVITEs over UDP to be dropped for the duration of the incident. **Technical Details** The SIP load balancers use Virtual Network Interface Cards \(VNICs\) to enable high availability. VNICs allow the public IP to remain unchanged while traffic is redirected to different physical instances \(e.g., during pod failures, software updates, or maintenance\). This supports seamless failover to standby instances and instance cycling. On March 2, 2026, an instance cycling process was initiated to apply security patches. Standard procedure involves adding additional IPs to DNS before rotating instances to maintain capacity. However, an operator error led to direct patching without first scaling up capacity. This operation should have been safe, as the VNIC is expected to redirect traffic to the new instance. * For TCP traffic, failover worked as intended. * For UDP, the VNIC's conntrack behavior differed: UDP flows are tracked using a four-tuple \(source IP:port - destination IP:port\). New incoming SIP INVITEs from the same client IP/port combination were treated as belonging to the same pre-failover "connection" causing them to be forwarded to the old node—even after it was terminated. * This issue specifically affected traffic from providers like Twilio, whose clients reused the same source IP/port for sequential INVITEs, resulting in dropped inbound calls. **Timeline** * 2026-03-02 10:30:00 UTC: Operator initiated instance cycling. * 2026-03-02 10:55:00 UTC: Monitoring systems detected the issue and paged the on-call engineer. * 2026-03-02 11:27:00 UTC: Traffic was redirected away from the affected US East region, mitigating the impact. **Monitoring & Detection** LiveKit employs two layers of end-to-end monitoring for the SIP infrastructure: 1. Simulated SIP pings: These send periodic OPTIONS packets to verify reachability of the SIP load balancers. They failed to detect the issue because each ping used a different source IP/port, avoiding the problematic conntrack entries. 2. End to end SIP calls: These use external providers \(e.g., Twilio\) to place real calls, verifying successful establishment and bidirectional audio flow. They run at a lower frequency. They successfully detected the drops \(as Twilio traffic matched the faulty conntrack behavior\), but detection was delayed due to the lower check interval. **Mitigations** To prevent recurrence and improve resilience, we will: * Increase redundancy at the SIP load balancer layer to tolerate up to 2 out of 3 nodes failing without service impact. * Enforce stricter, standardized operating procedures for instance cycling \(e.g., mandatory capacity addition via DNS updates before rotation; additional peer review or automation safeguards\). * Increase the frequency of end-to-end SIP call monitoring to enable faster detection of UDP-specific issues. This incident highlights subtle differences in how stateful conntrack handles UDP vs TCP during VNIC failover for protocols like SIP. These changes will help ensure more robust handling of similar maintenance operations in the future

**Root Cause** This issue had the same root cause as the [incident](https://status.livekit.io/incidents/xkcpnmycy5m3) that occurred after it in the US east region. Please view that link for a better understanding of the root cause, technical details, monitoring and mitigations. **Timeline** 2026-02-20 19:00 UTC – Isolated customer reports came in of no sip responses to invites 2026-02-20 20:34 UTC – Looking at various customer examples, we figured out that the common factor in problematic calls was the chicago based SIP loadbalancer. 2026-02-20 21:26 UTC – SIP in Chicago was drained after ensuring it there was enough capacity in other regions.

## Summary This incident was reported directly by customers and was not detected by our internal monitoring systems. Customer-reported issues of this nature are particularly concerning because they indicate gaps in our observability. Our monitoring failed to identify the failure and therefore did not trigger automated alerting or incident response. We sincerely apologize to affected customers and take this detection failure very seriously. We are committed to doing better. ## Root cause The root cause of connection hanging was due to degradation in two of our Network Load Balancers \(NLB\) in US East, resulting in a percentage of incoming HTTPS connections hanging before reaching our backend servers. Most of our client SDKs and applications \(Web, mobile, Go SDK, etc.\) have built-in timeouts and retry in order to survive failure modes like this. For these clients, a hanging initial connection would typically timeout quickly, followed by successful retries on subsequent attempts, effectively masking the underlying problem for the majority of users. The Rust SDK \(and other SDKs built on the Rust core\) was impacted much more severely. While it implements retries, it did not enforce a connection timeout on the initial attempt. This allowed connections to hang for much longer in affected cases, leading to noticeable stalls and degraded user experience for Rust-based clients. The primary reason this incident evaded detection was that our end-to-end monitoring included probes using the JavaScript and Go SDKs, both of which gracefully handled the hanging connections via timeouts and retries. This created a blind spot for the specific failure mode. ## Incident timeline \(all times in UTC, 2026-02-14\) 17:00 - Received report from customer that a percentage of connections were unsuccessful 17:10 - We started to investigate the reports 17:20 - We've confirmed that our connection tests are passing, and error/warning rate do not look elevated 17:25 - Team concludes \(prematurely\) that no widespread issue exists 20:45 - Direct testing by IP confirms that two NLBs in US East are hanging on a ~8% of requests 20:50 - On-call engineer paged to troubleshoot load balancers 22:00 - Traffic fully diverted from the degraded load balancers; service recovered 24:00 - Created new load balancers to replace the faulty ones ## Corrective Actions & Prevention The following improvements have been implemented or initiated to reduce the likelihood and impact of similar incidents: * Added proper connection timeouts to initial requests: [https://github.com/livekit/rust-sdks/pull/895](https://github.com/livekit/rust-sdks/pull/895) * Added dedicated, synthetic continuous monitoring for every individual load balancer \(health checks that are independent of SDK retry behavior\) * Opened a detailed root-cause investigation with our cloud provider regarding the NLB degradation. We are working with them to improve upstream detection, telemetry, and handling of similar failure modes. We will continue to expand failure-mode-specific monitoring \(beyond SDK-based probes\) and periodically validate that our alerting covers realistic client behaviors across all major SDKs. Thank you for your patience and understanding. We appreciate any additional feedback from customers who were affected.