LiveKit Outage History

Started Dec 28, 2025, 12:29 AM UTC· resolved Dec 28, 2025, 12:37 AM UTC

1. investigating Dec 28, 2025, 12:29 AM UTC

   we are investigating reports of connection failures originating from the Chicago region.

2. identified Dec 28, 2025, 12:34 AM UTC

   the error is due to an expired certificate in the US Central cluster, we are routing traffic around the region

3. resolved Dec 28, 2025, 12:37 AM UTC

   we've resolved the root cause of the expiring cert. service has been fully restored. this impacted core connectivity and API in the US Central (Chicago) region, including RTC, SIP, Phone numbers, Egress, and Ingress post mortem will follow shortly.

4. postmortem Dec 28, 2025, 06:22 PM UTC

   The service we use to auto-renew certificates throttles aggressively. When a request to renew a cert is throttled or failed, we automatically retry to renew the certificate. This process usually resolves and the cert is renewed. However, in some cases the renewal process can get stuck; we have alerting set up to catch these instances. Unfortunately a logic error was introduced into this alert and in this case a stuck renewal process did not alert and the certificate eventually expired. We have validated that no other certs are currently in this state. We have also resolved the issue with the alerting. In the future, we'll ensure to monitor changes made to this alerting process to ensure this does not happen again.

Read the full incident report →

Started Dec 11, 2025, 04:02 PM UTC· resolved Dec 12, 2025, 06:42 PM UTC

1. investigating Dec 12, 2025, 07:37 PM UTC

   We are currently investigating reports of increased API response times for Egress and SIP in our Frankfurt region.

2. resolved Dec 12, 2025, 11:49 PM UTC

   This incident has been resolved. p99 latency between services in the cluster and the DB instance in the eu-central region was elevated from a single cluster. Traffic has been routed around the cluster.

Read the full incident report →

Started Dec 07, 2025, 08:07 PM UTC· resolved Dec 08, 2025, 03:10 PM UTC

1. identified Dec 07, 2025, 08:07 PM UTC

   we are observing delays in the analytics data that's visible from the dashboard. we are working on processing the backlog of data. this impacts Session detail page, Egresses, Ingresses, and SIP dashboards. Sessions listing and Agent insights are not affected.

2. identified Dec 08, 2025, 08:29 AM UTC

   we've implemented fixes to help the data processor to catch up. live processing will be enabled soon.

3. resolved Dec 08, 2025, 03:10 PM UTC

   A fix has been implemented. The processing is back online, and will catch up to real time We will backfill the data for the affected duration next

Read the full incident report →

Started Dec 05, 2025, 09:00 AM UTC· resolved Dec 05, 2025, 09:30 AM UTC

1. resolved Dec 05, 2025, 09:37 AM UTC

   User traffic was not impacted, but our customer dashboard was not operational because of a major Cloudflare outage. It lasted roughly from 08:45 - 09:15 UTC. https://www.cloudflarestatus.com/incidents/k9ppxftx8bs5 has some details, though it appears the issue may have impacted Cloudflare caching as well. It appears Docker hub may have been impacted at the same time https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/6932a03b2276fb05ff7ee506 but at this point we see no production impact of this on Livekit.

Read the full incident report →

Started Dec 03, 2025, 03:00 PM UTC· resolved Dec 03, 2025, 03:00 PM UTC

1. resolved Dec 04, 2025, 05:46 AM UTC

   On 2025-12-03 from 15:06 to 15:26 UTC, network connectivity was flapping in our India, Israel, Saudi Arabia and UAE regions. During that period, attempts to initiate RTC connections or API requests to those regions would have failed intermittently. Clients failing to connect would have been routed to a nearby available region on a subsequent attempt. Clients would have experienced a slightly higher join time because of that.

Read the full incident report →

Started Nov 18, 2025, 07:42 PM UTC· resolved Nov 18, 2025, 07:42 PM UTC

1. resolved Nov 18, 2025, 07:42 PM UTC

   At 11:48 UTC a Cloudflare outage affected the ability to access LiveKit web pages, including www.livekit.io and our Cloud Dashboard (cloud.livekit.io). All other systems remained functional and were unaffected. End user services, including RTC, Ingress, Egress, Hosted Agents, and Inference Gateway remained functional throughout the outage. The issue was resolved by Cloudflare by 17:44 UTC.

Read the full incident report →

Started Nov 10, 2025, 04:29 PM UTC· resolved Nov 10, 2025, 06:30 PM UTC

1. investigating Nov 10, 2025, 04:29 PM UTC

   We are currently investigating reports of track egresses failing to start in our US West and US Central regions. Other egress types are not impacted.

2. resolved Nov 10, 2025, 08:47 PM UTC

   This incident was resolved as of 10:30 am PST and we have not observed any additional errors since then. We will update this incident with more details on the impact that we observed.

3. postmortem Nov 14, 2025, 12:55 AM UTC

   # Summary Track Egress experienced intermittent failures and delayed status updates caused by RPC instability between the egress and controller services in US regions \(primarily Phoenix and Chicago\). Room Composite egresses saw a much smaller impact. Other egress types were not broadly affected. Customer impact \(initial estimates during window 2025-11-10 14:00–18:30 UTC\): * ~2.25% of Track Egress start requests failed * ~0.0175% of Room Composite starts failed * ~0.75% of egresses had missing or delayed status updates The incident was caused by a bug in the egress RPC client causing RPCs to fail in come conditions, affecting status updates and egress service availability. # Timeline Timestamps are in UTC * 14:00 Some TrackEgress request start failing. Some successful egresses never reach the COMPLETE status in the cloud dashboard * 14:14: First alert for increased egress start latency. Investigation starts * 18:00: Issue is identified as RPC failures preventing some egress instances from updating the egress state other or servicing new egress requests * 18:15: New egress instances are brought up to replace the failed ones, mitigating the outage. Customer impact ends. * Nov 12: Underlying bug in RPC client is identified and deployed to the egress cluster. # Root Cause Analysis The LiveKit infrastructure relies on controller nodes to dispatch requests to egress nodes, and to update the stored Egress status. A RPC mechanism is used to transport messages between these 2 services. A bug in the egress RPC client caused it to rarely get into a bad state, preventing it from sending new RPC messages. This means that the egress instance would be unable to update an egress request status, or to start servicing new requests. This would in turn cause the egress cluster to run out of capacity. # Mitigations The failed egress instances were drained and replaced with new ones. These new instances were monitored to ensure thay did not get into the failed state. The underlying issue in the RPC implementation was identified and corrected. We also added a watchdog into egress instances to automatically replace them with new ones such an issue were to occur again.

Read the full incident report →

Started Nov 06, 2025, 03:39 PM UTC· resolved Nov 06, 2025, 10:03 PM UTC

1. investigating Nov 06, 2025, 03:39 PM UTC

   We are investigating intermittent issues with creating new cloud agents in EU Central. Running agents in this region are not affected.

2. identified Nov 06, 2025, 04:38 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Nov 06, 2025, 07:28 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 06, 2025, 10:03 PM UTC

   This incident has been resolved. An upstream service provider was having intermittent timeout issues. We've added retries to this interaction and we have not seen a reoccurrence.

Read the full incident report →

Started Oct 31, 2025, 12:37 PM UTC· resolved Oct 31, 2025, 02:35 PM UTC

1. investigating Oct 31, 2025, 12:37 PM UTC

   We are currently investigating the issue.

2. monitoring Oct 31, 2025, 01:04 PM UTC

   A fix has been implemented and we are monitoring the results.

3. identified Oct 31, 2025, 01:10 PM UTC

   The Reconciler service experienced an interruption, preventing certain changes (e.g., update-secret, deployments, etc.) from being applied to agents during the incident. Creation of new agents was not affected, and only a subset of existing agents were impacted.

4. monitoring Oct 31, 2025, 01:10 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Oct 31, 2025, 02:35 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 31, 2025, 09:05 AM UTC· resolved Oct 31, 2025, 09:05 AM UTC

1. resolved Oct 31, 2025, 09:05 AM UTC

   On 2025-10-31 from 06:42 to 06:55 UTC, we detected a cloud provider outage in Frankfurt. We automatically routed around this. After recovery, we noticed elevated RTT to other European clusters, so as a precaution we kept Frankfurt drained until the network proved stable. It was undrained at 09:08.

Read the full incident report →

Started Oct 30, 2025, 08:36 AM UTC· resolved Oct 30, 2025, 08:36 AM UTC

1. resolved Oct 30, 2025, 08:36 AM UTC

   Our Cloud Provider in Saudi Arabia went down from 7:30 - 8:20 UTC. We automatically route around these issues for unrestricted traffic, but customers with compliance requirements will have noticed a disruption of service.

Read the full incident report →

Started Oct 28, 2025, 06:26 PM UTC· resolved Oct 28, 2025, 08:15 PM UTC

1. identified Oct 28, 2025, 06:26 PM UTC

   We are investigating an issue with inbound SIP calls. Service should start to be restored soon

2. monitoring Oct 28, 2025, 06:34 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 28, 2025, 08:15 PM UTC

   This incident has been resolved. Inbound SIP has been fully recovered as of 11:38am PT and all metrics look normal. We'll be sharing a detailed post-mortem shortly.

4. postmortem Oct 29, 2025, 04:04 PM UTC

   ## Summary This incident was caused by a bug in our deployment stack. A bug in the deployment automation step unintentionally removed SIP load balancers from the deployment. This impacted inbound SIP calls globally for around 20 minutes. The load balancers were recreated once the team discovered the impact. ## Timeline Timestamps are in UTC 17:57 - a breaking change was merged in our deployment automation software. This change missed a Jsonnet generation step in our deployment automation. 18:01 - a code change was merged, triggering CI to deploy that change to staging. this step utilizes our deployment automation software. 18:09 - due to the missing Jsonnet generation, the deployment config showed that all Jsonnet controlled resources were scheduled to be removed \[first-outage start\] 18:11 - oncall was alerted to SIP load balancers failing pings 18:12 - incident was created internally to triage the issue 18:19 - we triaged the breakage to be caused by the deployment config change made at 18:09 18:23 - deployment config change has been reverted, inbound calls started recovering at this point \[first-outage end\] 18:25 - we’ve traced the breakage to the breakage in deployment automation code. 18:30 - another CI deploy merged in another config update with missing Jsonnet resource \[second-outage start\] 18:32 - upon discovery, we disabled CI internally so no other change would hit the same issue. 18:35 - the second config update was reverted \[second-outage end\] ## Root cause analysis We use an internal software stack to manage all of our software deployments across 21 clusters globally \(including staging and production\). The deployment software uses a couple of different templating engines in order to create the specific configuration for each service in every cluster. Jsonnet was introduced to the stack a couple of months ago, but it remains a fairly new component in the stack. The SIP load balancers were one of the services that utilized Jsonnet. When a code change was made to the deployment stack, it missed the Jsonnet generation step in the change. This caused the generated cluster-specific configuration to miss SIP load balancers entirely. Typically, any deployment that touches production resources is reviewed by an engineer, where the engineer looks at the diff of the change generated before approving the configuration deployment. In this case, while the change was targeted for staging, it impacted production resources due to the bug in the deployment software itself. The combination of the staging target and the bug caused the change to be pushed without engineering review. ## Mitigations After the incident, we performed an extensive review of our deployment systems and have identified a few shortcomings that we will address in the coming days. * improve error handling in the deployment stack, so the deployment is aborted if a missing dependency is found \(complete\) * CI-triggered deployment is only able to affect non-production resources * any production-related config change would need explicit engineering review * CI-triggered deployment with significant resource deletion will be disallowed by default

Read the full incident report →

Started Oct 22, 2025, 02:05 AM UTC· resolved Oct 22, 2025, 05:10 AM UTC

1. investigating Oct 22, 2025, 02:05 AM UTC

   We are currently investigating the issue. This affects only the analytics dashboard and api. Real time communications is operational

2. identified Oct 22, 2025, 02:10 AM UTC

   The issue has been identified. Dashboard is now accessible. We are working on restarting the processing pipeline

3. identified Oct 22, 2025, 02:10 AM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Oct 22, 2025, 04:49 AM UTC

   A fix has been implemented. We are continuing to monitor the situation

5. resolved Oct 22, 2025, 05:10 AM UTC

   This incident has been resolved. We will backfill/restore the data

Read the full incident report →