LiveKit Outage History

Started May 07, 2026, 05:16 PM UTC· resolved May 07, 2026, 09:15 PM UTC

1. resolved May 08, 2026, 08:25 PM UTC

   Between ~17:16 and 17:21 UTC on May 7, 2026, inter-region connectivity between our US East region and other regions was briefly degraded due to a network connectivity issue in our underlying cloud provider's data center. During this window, sessions that relied on cross-region track relay through US East may have experienced brief connection failures or reconnections for some participants. Our system recovered automatically once the cloud provider's data center network connectivity was restored.

Read the full incident report →

Started May 06, 2026, 07:46 PM UTC· resolved May 06, 2026, 11:04 PM UTC

1. investigating May 06, 2026, 07:46 PM UTC

   We are observing outbound calls using TLS with Twilio trunks fail and are investigating this urgently

2. monitoring May 06, 2026, 08:05 PM UTC

   A fix has been implemented and we are monitoring the results. The issue persisted from 18:10 - 19:58 UTC. We will follow up with a postmortem as soon as possible.

3. monitoring May 06, 2026, 08:32 PM UTC

   We've realized that this incident has impacted all outbound calls using TLS (not just those placed through Twilio) and have updated the incident title accordingly.

4. resolved May 06, 2026, 11:04 PM UTC

   We have observed no more errors since we pushed the fix. Resolving the incident.

5. postmortem May 10, 2026, 03:11 PM UTC

   **Summary** On 2026-05-06 between 18:07 and 19:56 UTC, a code change to the API server which handles CreateSIPParticipant calls caused outbound calls to incorrectly inherit their encryption settings from outbound trunks. This resulted in calls that didn't specify an encryption setting in the request itself to be sent without SRTP, even if the LiveKit trunk was configured to require it. If users had configured their carrier’s trunk to expect SRTP \(by enabling Twilio’s Secure Trunking, for example\) those INVITEs were rejected. Approximately 2% of global outbound SIP calls were affected during the incident window. **Impact** - Window: 2026-05-06 18:07 - 19:56 UTC - Scope: Outbound TLS calls where the encryption setting was not set in the request and the carrier was configured to expect SRTP. - Customer symptom: Outbound calls failed at the carrier with `488` rejections such as Twilio's `32208 SIP trunk or domain is required to use secure media (SRTP)`, `Not Acceptable Here`, or `Media Encryption Required`. - Identifying impact: Customers can search their call records during the window for failed outbound calls returning `488` from their carrier with one of the above reason phrases. **Root cause** A recent code change introduced a regression in how SIP trunk encryption settings were applied to outgoing calls. SIP trunks configured to require SRTP had that requirement silently dropped on outbound INVITEs when the per-call request did not also explicitly set an encryption mode. As a result, outbound calls were sent without SRTP, and carriers that were configured to expect SRTP on TLS correctly rejected them. **Timeline \(UTC\)** - 18:07 - Production deployment containing the regression went live. - 18:39 - Internal monitoring paged on a periodic end to end functionality test. - 19:42 - First customer-reported outbound call failures. - 19:56 - Fix deployed to production. Failure rate returned to baseline. **Resolution** A fix was deployed that ensures trunk-level SRTP requirements are correctly applied to outbound calls. Regression tests covering this and the adjacent code paths have been added to prevent the same class of issue. **What we are doing to prevent recurrence** - Strengthening pre-deployment validation requirements for changes that affect our SIP infrastructure. - Improving the reliability of our automated test coverage so similar regressions are caught before reaching production.

Read the full incident report →

Started May 06, 2026, 04:01 AM UTC· resolved May 06, 2026, 09:41 AM UTC

1. investigating May 06, 2026, 04:01 AM UTC

   We are currently investigating degraded performance on Agents Hosted on LiveKit Cloud in US East.

2. investigating May 06, 2026, 04:06 AM UTC

   We are currently investigating intermittent deployment failures on Cloud Agents in US East.

3. investigating May 06, 2026, 04:22 AM UTC

   New agent deployments are temporarily unavailable in the US East region, and our team is actively working on this. Dispatches to previously deployed agents are not affected.

4. investigating May 06, 2026, 05:00 AM UTC

   Our team is actively working to resolve deployment failures in US East. Existing deployed agents should not be affected in any way.

5. investigating May 06, 2026, 05:34 AM UTC

   etcd service in our US East Kubernetes cluster is currently down, resulting in API server unresponsiveness and failures for new deployments and redeployments. We are actively working on mitigating this with our data center provider.

6. investigating May 06, 2026, 08:47 AM UTC

   We are continuing to work on restoring the Kubernetes API server in US-east. Starting at 08:15 UTC, we are observing impact to existing agent deployments in addition to new deployments. Mitigation is in progress.

7. monitoring May 06, 2026, 09:17 AM UTC

   US-East recovered as of 08:55 UTC and has been serving traffic normally since. New and existing agent deployments are working as expected. We are moving to monitoring while we confirm sustained stability. A full post-incident review will follow.

8. resolved May 06, 2026, 09:41 AM UTC

   US-East Cloud Agents is now fully functional.

9. postmortem May 09, 2026, 07:36 AM UTC

   **Incident** On May 6, around 04:00 UTC we started to receive internal alarms about cluster availability from a single cluster in our us-east hosted agents region. Upon investigation we found that etcd was unresponsive for this cluster. Without etcd and control plane availability, new actions in the cluster were not processed. Anything already running previous to the start of the incident continued to run. However, builds, autoscaling and other similar events sent to this cluster failed. Around 04:30 UTC we escalated the issue to our cloud provider. Resolution took longer than expected for several reasons, including additional complexities in getting etcd and kube control plane back into a good state. We have multiple clusters in the region but we did not want to strain other clusters with the complete workload from the affected cluster. In order to ensure stability in other clusters we decided to add additional capacity to our fleet. Around 05:00 UTC we began setting up additional clusters and started making plans to migrate new deployments. Around 08:00 UTC a scale up operation began to revive the affected cluster. Around 09:15 UTC control plane and etcd resources began to recover. During the recovery process some existing workload became unstable, but our reconciler resolved the issue soon after. Around 09:40 UTC everything had recovered. **Post-incident** We've found better methods of escalation and recovery to ensure a similar delay doesn't happen in the future for cases like this. We've also been working on tuning our workloads so that a similar incident doesn't reoccur. We'll be adding additional monitoring and alarms for several specific scenarios that we've uncovered in our investigation. In addition, we're continuing our work to stand up additional compute resources so that individual cluster failures won't block things like deployments and scaling actions from completing. We've been working on a few initiatives along these lines already, but will be increasing the priority to ensure these items are completed soon.

Read the full incident report →

Started Apr 24, 2026, 06:12 PM UTC· resolved Apr 29, 2026, 02:41 PM UTC

1. investigating Apr 24, 2026, 06:12 PM UTC

   We received a single report of CreateSIPParticipant timeouts in Singapore. While investigating this, we discovered an increased rate of signalling connection errors on a very small minority of requests, mainly in India. We haven’t received any other reports, but are proactively creating this incident in case users encounter increased connection latency. We are continuing to investigate and will update here once we know more.

2. identified Apr 24, 2026, 11:20 PM UTC

   A fix is being implemented. We still have not received any further reports and believe the impact to be minor, but we're continuing to monitor for further issues.

3. resolved Apr 29, 2026, 02:41 PM UTC

   We are closing this incident as we have not received further reports and the signalling error rate has dropped back to baseline. We made some fine-tuning adjustments to our stack to improve performance and will continue to seek out opportunities to keep latency low.

Read the full incident report →

Started Apr 22, 2026, 01:00 PM UTC· resolved Apr 22, 2026, 01:00 PM UTC

1. resolved Apr 23, 2026, 01:38 AM UTC

   Between 13:15 and 14:25 UTC on April 22, approximately 5% of inbound SIP calls routed through our US East region were rejected by our system due to a failing internal trunk lookup. Upstream carriers surfaced these rejections to end users as 503 "Service Unavailable" responses. A recent change to our internal service responsible for SIP trunk authorization lookups caused trunk queries to return empty results under certain conditions. When our SIP service received an empty trunk lookup, it rejected the inbound INVITE. The regression was deployed to one US region as part of a staged rollout. Our routine checks during the release identified the issue. Once the offending change was rolled back, inbound call rejections returned to baseline within minutes and full service was restored. Other regions and all outbound calls were unaffected. We are introducing a dedicated monitor for this specific failure mode so that any recurrence pages our on-call engineers immediately, rather than relying on broader error-rate signals.

Read the full incident report →

Started Apr 15, 2026, 11:23 PM UTC· resolved Apr 16, 2026, 04:09 AM UTC

1. investigating Apr 15, 2026, 11:23 PM UTC

   We are investigating reports of increased latencies in RoomService APIs in the US West region, specifically on CreateRoom, DeleteRoom, and UpdateRoomMetadata APIs.

2. investigating Apr 16, 2026, 12:13 AM UTC

   We are continuing to investigate this issue.

3. investigating Apr 16, 2026, 01:07 AM UTC

   We believe these elevated latencies began around 22:00 UTC. We have confirmed that only API requests in US-West should be impacted. The current list of impacted APIs appears to be CreateRoom, DeleteRoom, and UpdateRoomMetadata. We are working on mitigating the issue to return latencies back to normal.

4. identified Apr 16, 2026, 02:10 AM UTC

   We continue to see the long Room API latencies which are now also impacting other regions. The latency increases appear to originate from a specific table in our distributed database. The issue has been escalated with the database vendor and we are working on a workaround for decreasing the API latencies. Other services are not impacted.

5. investigating Apr 16, 2026, 03:45 AM UTC

   While applying a fix for the API latencies, we are temporarily seeing increased failure rates in RoomServices APIs, including CreateRoom, UpdateRoomMetadata, and DeleteRoom. We are actively working on mitigating this. Impact has been upgraded to major.

6. monitoring Apr 16, 2026, 03:51 AM UTC

   Our fix is fully implemented and we are not seeing any more failures or high latencies of the RoomService APIs. We are continuing to monitor the issue. We did observe a period of 15 minutes with high API failures while mitigation steps were being applied.

7. resolved Apr 16, 2026, 04:09 AM UTC

   This issue is now fully resolved. We will be posting a detailed RCA.

8. postmortem Apr 17, 2026, 04:37 PM UTC

   ## Summary LiveKit's core realtime and agent services are designed to tolerate database failures. WebRTC media, SIP calls, and hosted agent sessions continue to operate even when our database backend is slow or unavailable. A subset of Room APIs, specifically `CreateRoom`, `DeleteRoom`, and `UpdateRoomMetadata`, do depend on a database for consistency and disaster recovery. That database is highly available and globally distributed, with no single point of failure. When it is under significant contention, these Room APIs can return errors or time out, while realtime traffic continues to flow normally. On 2026-04-15, database contention caused a percentage of Room API calls to fail in our US-West region. Remediation work later produced a 26-minute global outage of the Room APIs. Realtime sessions, SIP calls, and agent processes were unaffected throughout. We sincerely apologize to customers whose applications were disrupted during this incident. ## Impact The incident had two distinct phases of customer impact. ### Phase 1: Elevated Room API timeouts in US-West \(2026-04-15 22:10 UTC to 2026-04-16 03:14 UTC\) A percentage of `DeleteRoom`, `UpdateRoomMetadata`, and `ListRooms` calls timed out, primarily in our US-West region. Other regions saw limited impact during this phase. Customers with high Room API volume in US-West observed elevated error rates on their integrations; the majority of customers were not affected. ### Phase 2: Global Room API outage \(2026-04-16 03:14 to 03:40 UTC, ~26 minutes\) While we were swapping in a rebuilt `rooms` table, the table was briefly missing from the database, and the majority of Room API calls globally returned HTTP 500 with `ERROR: relation "rooms" does not exist`. WebRTC sessions, SIP calls, and agent processes continued to function, and realtime connection counts remained stable. Applications that depend on Room APIs to start or manage sessions saw visible failures during this window. ## Root Cause The sweeper is a background process that removes rows from the `rooms` table as sessions end. Earlier on 2026-04-15, its throughput dropped significantly, and over roughly 8 hours stale rows accumulated to the point where the table was many times larger than its intended steady-state size. At approximately 21:00 UTC, a routine schema migration was applied to a different, unrelated table. The migration itself did not touch `rooms`, but it raised overall database disk utilization and background load. Combined with the oversized `rooms` table, this produced enough contention to slow down reads and writes against it. The effect first appeared in US-West, where the regional mix of Room API traffic was most sensitive to the contention. Once we identified the oversized table as the underlying cause, we needed to restore it to a healthy size. Because the table was already contended, deleting rows directly would have taken additional locks and worsened the contention. We instead chose to rebuild the table: create a new table with the same schema, copy over the active rows, then atomically swap the new table into place via a pair of renames. The copy phase completed quickly. The first rename \(moving the old `rooms` table aside\) completed in about 2.5 minutes. The second rename, moving the new table into the `rooms` name, stalled on our globally distributed database for significantly longer than we anticipated. During the stall, the `rooms` table did not exist from the perspective of any region, and all Room API calls globally returned errors. After roughly 10 minutes, we aborted the stalled rename, created a fresh `rooms` table from scratch, and inserted the active rows into it. Room API traffic recovered globally shortly after. ## Corrective Actions & Prevention The following improvements have been implemented or initiated to reduce the likelihood and impact of similar incidents: * **Enhance monitoring for sweeper throughput and active room count.** We are adding and hardening alerts on sweeper throughput and active room count, so that any future divergence pages on-call well before it threatens production. * **Improve sweeper resilience and throughput.** We are investigating the cause of the sweeper's throughput drop and adding capacity headroom so a transient slowdown cannot translate into multi-hour backlog growth. * **Remove database as a dependency for Room APIs**. This incident reaffirmed our long-held design principle that realtime services should not depend on databases. We believe this is the only way to build a system that approaches 100% uptime, and we will continue the work to ensure Room APIs do not depend on a database either. The Phase 2 outage was caused by our own remediation, and we recognize how disruptive it was for applications that depend on the Room APIs. We are committed to the work above to reduce both the likelihood and the blast radius of a similar failure in the future. Thank you for your patience, and we welcome any additional feedback from customers who were affected.

Read the full incident report →

Started Apr 08, 2026, 04:18 PM UTC· resolved Apr 08, 2026, 04:18 PM UTC

1. resolved Apr 08, 2026, 04:18 PM UTC

   We have identified a bug where LiveKit Cloud projects created after April 1 at 20:41 UTC which enabled Agent Insights encountered a bug where their observability data (recordings, traces, and agent logs) was not saved correctly. We have resolved the issue on April 8 at 03:19 UTC and Agent Insights should be fully operational for all new and existing projects. No action is required from impacted users. We will follow up with a postmortem as soon as possible.

Read the full incident report →

Started Apr 03, 2026, 05:13 PM UTC· resolved Apr 03, 2026, 05:23 PM UTC

1. investigating Apr 03, 2026, 05:13 PM UTC

   We are currently investigating this issue and will update as soon as we know more.

2. resolved Apr 03, 2026, 05:23 PM UTC

   A fix has been implemented and we are monitoring the results. All users should now be able to access the Cloud Dashboard.

3. postmortem Apr 03, 2026, 11:00 PM UTC

   **Root Cause** A gap in our DNS update process resulted in a misconfiguration of the DNS for [livekit.io](http://livekit.io), causing several A records to be overwritten. This resulted in an incorrect IP address being returned for [livekit.io](http://livekit.io). All real-time services - calls, RTC, and hosted agents - were unaffected as they operate on separate domains. We do have DNS monitoring in place, but it was not configured to page on-call. As a result, the issue was identified by LiveKit engineering after a short delay. **Timeline** * **2026-04-03 17:46 UTC**: DNS configuration change applied; resolution begins failing for [livekit.io](http://livekit.io). * **2026-04-03 17:52 UTC**: Issue identified by LiveKit. * **2026-04-03 18:15 UTC**: Root cause identified and fix applied. * **2026-04-03 18:18 UTC**: Fix fully propagated; services restored. Total duration: ~32 minutes. **Mitigations** * Enforce tighter automated guardrails around DNS updates. * Enable paging on existing DNS health checks so any future issues are caught even sooner.

Read the full incident report →

Started Apr 01, 2026, 10:37 AM UTC· resolved Apr 01, 2026, 01:01 PM UTC

1. monitoring Apr 01, 2026, 10:37 AM UTC

   We identified degraded connection errors affecting a small percentage of requests to Real Time Communication and TURN services in our EU (Frankfurt) region. Automatic retries prevented applications from experiencing failures. A fix has been implemented and we are actively monitoring to confirm stability.

2. resolved Apr 01, 2026, 01:01 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 30, 2026, 11:26 AM UTC· resolved Mar 30, 2026, 09:01 PM UTC

1. investigating Mar 30, 2026, 11:26 AM UTC

   We are currently investigating the issue. Only dashboard updates are affected (real time communication is not affected).

2. identified Mar 30, 2026, 03:01 PM UTC

   The issue has been identified. We are working on a fix.

3. identified Mar 30, 2026, 06:46 PM UTC

   A fix has been deployed. We expect the service to be restored shortly and will provide another update once it is back online.

4. monitoring Mar 30, 2026, 08:12 PM UTC

   The fix has been applied and real-time processing for the Cloud Dashboard is back online globally. We are monitoring the pipeline.

5. monitoring Mar 30, 2026, 08:55 PM UTC

   We have initiated the process of backfilling delayed analytics data.

6. resolved Mar 30, 2026, 09:01 PM UTC

   Real-time analytics updates on Cloud Dashboard is now fully functional, and delayed data are in the process of being backfilled.

Read the full incident report →

Started Mar 30, 2026, 09:09 AM UTC· resolved Mar 30, 2026, 09:45 AM UTC

1. investigating Mar 30, 2026, 09:09 AM UTC

   Users may be unable to access the dashboard. Our team is actively investigating.

2. investigating Mar 30, 2026, 09:09 AM UTC

   We are continuing to investigate this issue.

3. monitoring Mar 30, 2026, 09:20 AM UTC

   A fix has been implemented and we are monitoring the result

4. resolved Mar 30, 2026, 09:45 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 23, 2026, 06:00 PM UTC· resolved Mar 23, 2026, 06:00 PM UTC

1. resolved Mar 24, 2026, 01:17 AM UTC

   Between 15:00–17:25 UTC, a subset of SIP outbound calls in the US East region experienced failures due to an internal DNS resolution issue on a single host. Approximately 1.22% of calls within the region were affected. The affected host was taken out of rotation and workloads were redistributed. Call failure rates returned to baseline following remediation.

Read the full incident report →

Started Mar 18, 2026, 06:30 PM UTC· resolved Mar 18, 2026, 06:30 PM UTC

1. resolved Mar 18, 2026, 09:04 PM UTC

   A dependency update in our latest docs deployment caused all requests to the MCP server at docs.livekit.io/mcp/ to return 500 errors. The issue began at 18:50 UTC and ended at 19:33 UTC. During this roughly 45 minute window, MCP clients - including IDE integrations and the LiveKit CLI - were unable to connect to the docs MCP server. The issue was caused by a package upgrade that introduced dependencies incompatible with our serverless hosting environment. We rolled back the change and confirmed the endpoint was fully operational at 19:33 UTC. No other services were affected.

Read the full incident report →

Started Mar 18, 2026, 11:25 AM UTC· resolved Mar 18, 2026, 11:25 AM UTC

1. resolved Mar 18, 2026, 11:25 AM UTC

   We experienced degraded networking affecting core real time communication services in Mumbai region. The issue began at 06:06 UTC and was resolved by 06:22 UTC on March 18, 2026. All services are now restored and we are continuing to monitor.

Read the full incident report →

Started Mar 13, 2026, 08:00 PM UTC· resolved Mar 13, 2026, 08:00 PM UTC

1. resolved Mar 14, 2026, 12:56 AM UTC

   Between 21:00 and 21:15 UTC, a limited amount of participant subscriptions failed across regions, along with increased subscription latency to subscribers in the Chicago region

Read the full incident report →

Started Mar 02, 2026, 10:30 AM UTC· resolved Mar 02, 2026, 10:30 AM UTC

1. resolved Mar 02, 2026, 12:22 PM UTC

   SIP calling over UDP in the US East region experienced failures from ~10:30–11:35 UTC, affecting inbound calls from Twilio. TCP and TLS traffic was not impacted. Service has recovered and we are monitoring.

2. postmortem Mar 06, 2026, 09:15 AM UTC

   **Root Cause** During a failover event on the SIP load balancer in US East, UDP packets were incorrectly treated as part of an existing "connection" by the underlying VNIC stack's connection tracking \(conntrack\) mechanism. As a result, these packets continued to be forwarded to the previous \(now non-existent\) node, causing SIP INVITEs over UDP to be dropped for the duration of the incident. **Technical Details** The SIP load balancers use Virtual Network Interface Cards \(VNICs\) to enable high availability. VNICs allow the public IP to remain unchanged while traffic is redirected to different physical instances \(e.g., during pod failures, software updates, or maintenance\). This supports seamless failover to standby instances and instance cycling. On March 2, 2026, an instance cycling process was initiated to apply security patches. Standard procedure involves adding additional IPs to DNS before rotating instances to maintain capacity. However, an operator error led to direct patching without first scaling up capacity. This operation should have been safe, as the VNIC is expected to redirect traffic to the new instance. * For TCP traffic, failover worked as intended. * For UDP, the VNIC's conntrack behavior differed: UDP flows are tracked using a four-tuple \(source IP:port - destination IP:port\). New incoming SIP INVITEs from the same client IP/port combination were treated as belonging to the same pre-failover "connection" causing them to be forwarded to the old node—even after it was terminated. * This issue specifically affected traffic from providers like Twilio, whose clients reused the same source IP/port for sequential INVITEs, resulting in dropped inbound calls. **Timeline** * 2026-03-02 10:30:00 UTC: Operator initiated instance cycling. * 2026-03-02 10:55:00 UTC: Monitoring systems detected the issue and paged the on-call engineer. * 2026-03-02 11:27:00 UTC: Traffic was redirected away from the affected US East region, mitigating the impact. **Monitoring & Detection** LiveKit employs two layers of end-to-end monitoring for the SIP infrastructure: 1. Simulated SIP pings: These send periodic OPTIONS packets to verify reachability of the SIP load balancers. They failed to detect the issue because each ping used a different source IP/port, avoiding the problematic conntrack entries. 2. End to end SIP calls: These use external providers \(e.g., Twilio\) to place real calls, verifying successful establishment and bidirectional audio flow. They run at a lower frequency. They successfully detected the drops \(as Twilio traffic matched the faulty conntrack behavior\), but detection was delayed due to the lower check interval. **Mitigations** To prevent recurrence and improve resilience, we will: * Increase redundancy at the SIP load balancer layer to tolerate up to 2 out of 3 nodes failing without service impact. * Enforce stricter, standardized operating procedures for instance cycling \(e.g., mandatory capacity addition via DNS updates before rotation; additional peer review or automation safeguards\). * Increase the frequency of end-to-end SIP call monitoring to enable faster detection of UDP-specific issues. This incident highlights subtle differences in how stateful conntrack handles UDP vs TCP during VNIC failover for protocols like SIP. These changes will help ensure more robust handling of similar maintenance operations in the future

Read the full incident report →

Started Feb 27, 2026, 12:18 AM UTC· resolved Feb 27, 2026, 02:40 AM UTC

1. identified Feb 27, 2026, 12:18 AM UTC

   Updates to the LiveKit Cloud analytics dashboards are currently delayed due to an issue with our processing pipeline. Ingestion of new data is not affected, and we are in the process of recovering all data. We'll share another update once the restoration is complete and dashboards are updating normally.

2. resolved Feb 27, 2026, 02:40 AM UTC

   Analytics dashboards are updating normally again.

Read the full incident report →

Started Feb 20, 2026, 09:47 PM UTC· resolved Feb 21, 2026, 04:54 AM UTC

1. investigating Feb 20, 2026, 09:47 PM UTC

   We are currently investigating user reports of SIP invites not getting responses in our Chicago region. We have not yet determined if the issue is due to the users' trunking providers, but we have begun routing traffic away from Chicago to our other US clusters to ensure continued service.

2. monitoring Feb 20, 2026, 10:41 PM UTC

   The users who made the original reports are no longer seeing issues. We will continue investigating and post the results of our investigation as soon as they are available, but there are currently no known issues in LiveKit's SIP infrastructure.

3. resolved Feb 21, 2026, 04:54 AM UTC

   We are going to keep traffic routing to our other US clusters, but we will consider this incident closed while we continue to investigate its origin as there is currently no impact to users.

4. postmortem Mar 06, 2026, 09:20 AM UTC

   **Root Cause** This issue had the same root cause as the [incident](https://status.livekit.io/incidents/xkcpnmycy5m3) that occurred after it in the US east region. Please view that link for a better understanding of the root cause, technical details, monitoring and mitigations. **Timeline** 2026-02-20 19:00 UTC – Isolated customer reports came in of no sip responses to invites 2026-02-20 20:34 UTC – Looking at various customer examples, we figured out that the common factor in problematic calls was the chicago based SIP loadbalancer. 2026-02-20 21:26 UTC – SIP in Chicago was drained after ensuring it there was enough capacity in other regions.

Read the full incident report →

Started Feb 14, 2026, 04:00 PM UTC· resolved Feb 14, 2026, 04:00 PM UTC

1. resolved Feb 20, 2026, 09:12 AM UTC

   We received customer reports indicating that some agent connections to LiveKit Cloud were timing out. The symptoms typically appeared as a networking error, with calls hanging indefinitely on `room.connect()` Investigation confirmed the root cause: degradation on two Network Load Balancers (NLBs) in the US-East region. This issue affected approximately 8% of inbound agent connections to US-East during the impacted period.

2. postmortem Feb 20, 2026, 09:16 AM UTC

   ## Summary This incident was reported directly by customers and was not detected by our internal monitoring systems. Customer-reported issues of this nature are particularly concerning because they indicate gaps in our observability. Our monitoring failed to identify the failure and therefore did not trigger automated alerting or incident response. We sincerely apologize to affected customers and take this detection failure very seriously. We are committed to doing better. ## Root cause The root cause of connection hanging was due to degradation in two of our Network Load Balancers \(NLB\) in US East, resulting in a percentage of incoming HTTPS connections hanging before reaching our backend servers. Most of our client SDKs and applications \(Web, mobile, Go SDK, etc.\) have built-in timeouts and retry in order to survive failure modes like this. For these clients, a hanging initial connection would typically timeout quickly, followed by successful retries on subsequent attempts, effectively masking the underlying problem for the majority of users. The Rust SDK \(and other SDKs built on the Rust core\) was impacted much more severely. While it implements retries, it did not enforce a connection timeout on the initial attempt. This allowed connections to hang for much longer in affected cases, leading to noticeable stalls and degraded user experience for Rust-based clients. The primary reason this incident evaded detection was that our end-to-end monitoring included probes using the JavaScript and Go SDKs, both of which gracefully handled the hanging connections via timeouts and retries. This created a blind spot for the specific failure mode. ## Incident timeline \(all times in UTC, 2026-02-14\) 17:00 - Received report from customer that a percentage of connections were unsuccessful 17:10 - We started to investigate the reports 17:20 - We've confirmed that our connection tests are passing, and error/warning rate do not look elevated 17:25 - Team concludes \(prematurely\) that no widespread issue exists 20:45 - Direct testing by IP confirms that two NLBs in US East are hanging on a ~8% of requests 20:50 - On-call engineer paged to troubleshoot load balancers 22:00 - Traffic fully diverted from the degraded load balancers; service recovered 24:00 - Created new load balancers to replace the faulty ones ## Corrective Actions & Prevention The following improvements have been implemented or initiated to reduce the likelihood and impact of similar incidents: * Added proper connection timeouts to initial requests: [https://github.com/livekit/rust-sdks/pull/895](https://github.com/livekit/rust-sdks/pull/895) * Added dedicated, synthetic continuous monitoring for every individual load balancer \(health checks that are independent of SDK retry behavior\) * Opened a detailed root-cause investigation with our cloud provider regarding the NLB degradation. We are working with them to improve upstream detection, telemetry, and handling of similar failure modes. We will continue to expand failure-mode-specific monitoring \(beyond SDK-based probes\) and periodically validate that our alerting covers realistic client behaviors across all major SDKs. Thank you for your patience and understanding. We appreciate any additional feedback from customers who were affected.

Read the full incident report →

Started Jan 28, 2026, 04:45 PM UTC· resolved Jan 28, 2026, 09:45 PM UTC

1. investigating Jan 28, 2026, 04:45 PM UTC

   We are investigating reports of SIP INVITE requests not receiving responses in some EU regions. The impact appears to be very low. We will provide further details as they become available.

2. monitoring Jan 28, 2026, 05:50 PM UTC

   We identified an issue in the UK related to inbound calls with large UDP INVITE messages that exceed the MTU. This affected 0.04% of inbound calls in the region. We have updated a configuration for handling fragmented packets in that cluster, and the issue was mitigated as of 16:10 UTC. The impact was limited to the UK; this issue was not observed in other clusters.

3. investigating Jan 28, 2026, 07:32 PM UTC

   We are continuing to investigate this issue.

4. investigating Jan 28, 2026, 07:46 PM UTC

   This issue is affecting customers sending UDP SIP traffic that exceeds the standard SIP MTU size. Switching to TCP is an immediate way to mitigate the problem.

5. identified Jan 28, 2026, 08:09 PM UTC

   We have identified the source of the issue and are deploying a fix.

6. monitoring Jan 28, 2026, 08:22 PM UTC

   A fix has been deployed and we are monitoring the results.

7. resolved Jan 28, 2026, 09:45 PM UTC

   This incident has been resolved. We will share a postmortem shortly.

8. postmortem Jan 31, 2026, 12:17 AM UTC

   Typically, SIP INVITE messages fit within the standard Ethernet MTU. In cases where a large number of headers or metadata are included, an INVITE can exceed the MTU and be fragmented. While such large messages are ideally sent over TCP, some providers may still send them over UDP. On 2026-01-08 06:11 UTC, we deployed updated firewall rules in our UK and EU regions as part of our regular security review. During this update, a configuration that allows fragmented packets was unintentionally omitted. As a result, larger SIP INVITE messages \(inbound calls\) sent over UDP were dropped. This issue affected a limited subset of customers in the UK and EU regions, accounting for approximately 1.5% of total inbound calls in the affection regions during the highest point of traffic for the day. The issue was identified and the firewall rules were rolled back at 2026-01-08 20:17 UTC. We already maintain end-to-end tests that cover large, fragmented packets. However, these tests were previously run only in a subset of regions. To prevent similar incidents in the future, we are expanding these tests to run in every cluster and to execute immediately after firewall and security configuration updates are applied.

Read the full incident report →

Started Jan 23, 2026, 02:39 PM UTC· resolved Jan 23, 2026, 10:24 PM UTC

1. investigating Jan 23, 2026, 02:39 PM UTC

   We are currently investigating reports of elevated API latency in the UK region.

2. identified Jan 23, 2026, 02:50 PM UTC

   We are routing traffic away from the London cluster to mitigate impact.

3. identified Jan 23, 2026, 04:21 PM UTC

   We are continuing to investigate the root cause of the performance degradation of the London cluster. We will continue to share updates as we learn more.

4. monitoring Jan 23, 2026, 06:38 PM UTC

   All non-pinned traffic has been healthy since we began routing traffic through the nearest healthy nodes at 14:50 UTC. Our team is still working to bring the London cluster back to full health.

5. resolved Jan 23, 2026, 10:24 PM UTC

   We brought the London cluster back to full health at 21:40 UTC. This incident is now resolved. We will publish a postmortem outlining the root cause and the steps we will take to ameliorate this issue in the future as soon as possible.

6. postmortem Jan 27, 2026, 09:35 PM UTC

   Our cloud provider’s managed Kubernetes control plane experienced an issue that caused connectivity to cluster nodes to be disrupted, dropping some of the workloads in the cluster. Due to a bug in our auto-draining logic, detection of the issue was delayed. We have since fixed the alerting to ensure faster detection and response if a similar situation occur should occur in the future.

Read the full incident report →

Started Jan 22, 2026, 06:29 PM UTC· resolved Jan 22, 2026, 07:46 PM UTC

1. identified Jan 22, 2026, 06:29 PM UTC

   We're investigating an issue with a subset of inference calls made to gemini models.

2. monitoring Jan 22, 2026, 06:34 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 22, 2026, 07:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 09, 2026, 10:00 AM UTC· resolved Jan 09, 2026, 10:24 AM UTC

1. investigating Jan 09, 2026, 10:00 AM UTC

   We are investigating customer reports regarding SIP outbound calls. Inbound calls are not affected. We'll provide more details soon. (SIP Global)

2. investigating Jan 09, 2026, 10:23 AM UTC

   We have mitigated the issue by rolling back a media service release that went out around the same time. The time of impact was about 07:41 - 10:01 UTC. (SIP Global)

3. investigating Jan 09, 2026, 10:24 AM UTC

   We are continuing to investigate this issue. (SIP Global)

4. resolved Jan 09, 2026, 10:24 AM UTC

   This incident has been resolved. It is our highest priority to understand the root cause and will share a full post mortem shortly

5. postmortem Jan 10, 2026, 07:14 PM UTC

   **Root Cause** A code change unintentionally overrode the user-supplied `ringing_timeout` for synchronous `CreateSIPParticipant` API calls. As a result, calls with `wait_until_answered=true` timed out significantly earlier than intended and failed prematurely. Widespread impact lasted for over 2 hours because it evaded our automated tests and monitoring systems. Below, we share some technical details behind the failure to provide transparency. **Technical Details** The `CreateSIPParticipant` API, used for making outbound calls, supports two modes of operation: * Async mode: This is the default behavior, where the API dials the call and returns `200` immediately. * Sync mode: When `wait_until_answered` is set to `true`, the API holds the connection open until the user answers or declines the call. In sync mode, the API returns a `200` status only if the user answers the call, or a `408` if they do not respond within the user-defined `ringing_timeout` \(which defaults to 30 seconds\). In an effort to enhance observability in our RPC stack, we've recently introduced a change to enable tracing for internal RPC calls. This automatically exports traces to our observability platform whenever an internal RPC is invoked. However, enabling this tracing also unintentionally introduced a default timeout of 3 seconds on internal RPCs. Consequently, two competing timeouts came into play during `CreateSIPParticipant` calls: * `ringing_timeout`: User-defined, defaults to 30 seconds. * internal RPC timeout: Fixed at 3 seconds. These inconsistent timeouts caused internal APIs to return a `408` error before the `ringing_timeout` was reached. As a result, SIP outbound calls with `wait_until_answered=true` would ring for only three seconds before aborting. Calls answered within 3 seconds or those in async mode proceeded without issues. **Detection and Response Challenges** Service reliability is our top priority; we maintain rigorous testing and alerting systems, including: * Continuous end-to-end tests running against both staging and production environments. * Phased deployment across our global infrastructure, starting with low-traffic regions. * Alarms triggered by high error rates \(5xx\) on customer-facing API calls. * Alarms for elevated internal RPC error rates. * Manual review of key health indicators during deployments. Despite these measures, the issue went undetected during deployment for the following reasons: * The 3s timeout has caused `CreateSIPParticipant` to return a `408`, before it has reached `ringing_timeout`. * Initial rollout regions had insufficient users relying on sync mode, so their calls completed without disruption. * Our end-to-end tests simulate actual calls but use a bot on the receiving end, which answers within 3 seconds. * SIP health indicators showed calls being made and completing overall \(though average call duration dropped during the incident, it was not monitored as a key health metric\). **Timeline** 2026-01-08 04:47 UTC – Change first deployed to a limited set of low-traffic regions. 2026-01-08 15:40 UTC – Second rollout phase to regions including Asia. 2026-01-08 20:45 UTC – Third rollout phase to additional regions, including EU. 2026-01-09 07:38 UTC – Change deployed to the majority of regions, resulting in widespread impact. 2026-01-09 09:58 UTC – Change fully rolled back, resolving the issue. **Scope of Impact** During the incident window, outbound calls meeting the following conditions failed: * `wait_until_answered=true ` * User did not answer within 3s **Mitigations and Follow-ups** To prevent similar issues in the future, we are implementing the following: * A more robust design for managing internal and system-level timeouts, scheduled for rollout within the next week. * Updates to end-to-end testing to include scenarios with longer delays before call pickup. * Addition of call duration as a key health indicator in our monitoring dashboards. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions or feedback, please reach out to our support team.

Read the full incident report →

Started Jan 07, 2026, 04:00 PM UTC· resolved Jan 07, 2026, 04:00 PM UTC

1. resolved Jan 07, 2026, 08:40 PM UTC

   Between 16:01 UTC and 16:05 UTC, ~9-10% of participants connecting to the US East region experienced disconnection/reconnection to the RTC service. Client SDKs automatically reconnect and the majority of affected users' access was restored by 16:05 UTC. Cause A code change from a new feature was the root cause of the issue. The new feature is behind a flag which is disabled by default. But, changes to code to support the new feature affected an existing path that did not implement the feature flag, and triggered its use by some mobile phone clients. Resolution The deployment that caused the issue was rolled back. The erroneous code path is being fixed and more tests will be added to ensure that it exhibits expected behavior.

Read the full incident report →

Started Dec 31, 2025, 07:24 PM UTC· resolved Jan 01, 2026, 04:02 AM UTC

1. investigating Dec 31, 2025, 07:24 PM UTC

   Less than 5 percent of inbound calls are seeing delays and no response back to INVITEs. We are actively investigating this.

2. monitoring Dec 31, 2025, 09:00 PM UTC

   A code change has been pushed to mitigate the issue. We are currently monitoring

3. resolved Jan 01, 2026, 04:02 AM UTC

   We've added more stats to track if the issue is recurring and we have not seen this issue recur. We are working on the postmortem with timelines and will update it soon.

4. postmortem Jan 03, 2026, 07:01 PM UTC

   **Root Cause:** The issue was caused by an incorrectly formed header in BYE requests, which triggered an infinite forwarding loop of SIP messages on our SIP load balancing service. This resulted in the server repeatedly forwarding messages to itself, causing resource exhaustion on the affected machines. During these periods, processing of incoming INVITE requests were delayed on affected servers. **Time window of impact:** We saw the first instances of these delays happen at 2025-12-28 19:00:00 UTC. While we pushed the first patches to mitigate this issue to majorly impacted regions on 2026-01-01 04:00:00 UTC, the fix was rolled out globally on 2026-01-02 19:00:00 UTC. **Scope of impact:** The incident had a much lower impact vs the 5% assessment we made during the incident based on limited data. The incident is now categorized as degraded service as opposed to a partial outage. In terms of actual impact to calls, out of the small percentage of calls that rang longer \(stats below\), a smaller subset of them could have timed out depending on what the timeout settings of the inbound calling trunk were. Regions affected: Eastern US Daily impact \(percentage of calls that rang longer than 2s - all times in UTC\): 1. 2025-12-28: 0.076% 2. 2025-12-29: 0.061% 3. 2025-12-30: 0.103% 4. 2025-12-31: 0.0046% Worst impacted hourly windows \(percentage of calls that rang longer than 2s - all times in UTC\): 1. 2025-12-28 19:00 \+ 1h: 1.01% 2. 2025-12-29 18:00 \+ 1h: 0.32% 3. 2025-12-30 16:00 \+ 1h: 0.37% 4. 2025-12-31 19:00 \+ 1h: 0.38% **Mitigations:** * We already applied a quick fix to mitigate the issue over the past few days. * We will be rolling out code changes to handle this edge case more comprehensively by mid next week. * More aggressive monitoring to detect delays in “SIP 100 Trying” responses is being worked on.

Read the full incident report →