LiveKit Outage History

Started Oct 31, 2026, 07:10 AM UTC· resolved Oct 31, 2026, 08:49 AM UTC

1. resolved Oct 31, 2026, 07:43 AM UTC

   This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed. - About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed. The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC. Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.

2. postmortem Nov 03, 2026, 01:54 PM UTC

   **Summary** On October 31, 2026, between 07:10 and 07:43 UTC, an uncommon data pattern surfaced a latent software bug which caused media servers in several regions to restart repeatedly. A small percentage of participants were disconnected and the vast majority automatically reconnected. Impact was concentrated in North America. We stopped the trigger at 07:43 UTC and deployed a fix to all regions by 08:49 UTC. We apologize for the disruption this caused. **Root Cause** Applications can open their own data channel and send raw data over it, outside of LiveKit's standard data message format. Media servers forward this raw data to other participants in the room, wrapping it with the sender's identity so that receivers know who sent it. When a room spans multiple regions, the media servers hosting it also forward this data to each other. That server-to-server step sent the raw data without the sender's identity, and the receiving server assumed the identity would be present, which caused it to crash. This gap in our server-to-server protocol had been present for some time and was not introduced by a recent deployment. Starting at 07:10 UTC, one application sent raw data over its own data channel in a room that spanned multiple regions. Every server that received the forwarded message crashed at the same time, while the server the sender was connected to kept running and kept sending. Participants on the crashed servers automatically reconnected to other servers. Those servers then began hosting the room, received the next message, and crashed as well. This cycle repeated until we stopped the trigger at 07:43 UTC. In total, media servers in 12 of our data centers restarted, some of them up to 10 times. **Timeline \(UTC\)** 2026-10-31 07:10 - An application begins sending raw data over its own data channel in a multi-region room. Media servers in US East and Canada restart within seconds. 07:13 - Our monitoring detects the restarts and an incident is opened automatically. 07:10 to 07:43 - Media servers in 12 data centers restart repeatedly as participants reconnect to new servers. 07:43 - We block the traffic triggering the bug. There are no further restarts after this point, and customer impact ends. 07:46 - A fix for the bug is ready. 08:04 - The fix begins rolling out, starting with a small set of regions. 08:49 - The fix is deployed to all regions. **Scope of Impact** Realtime sessions: * Across all regions, about 1.5% were disconnected more than once within five minutes. * In US East, US Central, US West, Canada and Mexico, about 6% were disconnected more than once within five minutes. * The median time from disconnect to rejoin was about 3 seconds. Agents: * Agents were disconnected and reconnected the same way as other participants, with the same median reconnect time of about 3 seconds. An agent that reconnects keeps its conversation state, so most conversations continued after a short gap. Fewer than 1% of active agent sessions did not recover. Ingress, Egress and SIP: * Most active WHIP ingress streams were interrupted, and reconnects typically took about 40 seconds. Some WHIP ingress update and delete requests also failed. * About 5% of active egress sessions were disconnected. Most reconnected within a few seconds. * About 1% of SIP participants were disconnected, and half of them rejoined in under a second. Some SIP call transfers also failed. APIs: * About 1.3% of Room Service API requests failed during the window. ListParticipants and GetParticipant were the most affected. How to tell if you were affected: participants in affected sessions saw a reconnecting state, or a brief loss of audio and video. In session analytics, an affected participant appears to leave and rejoin the same room with the same identity between 07:10 and 07:43 UTC. **Detection and Response** Our monitoring detected the server restarts about 3 minutes after they began, and an incident was opened automatically. Identifying the specific room and traffic triggering the restarts took about 30 minutes. Once we found it, we blocked the trigger at 07:43 UTC and the restarts stopped immediately. We deployed a fix to all regions by 08:49 UTC. We did not post to our status page while the incident was in progress, and published a notice only after it was resolved. Our original post mistakenly sent notifications as we initially over-estimated the scope of impact. **Mitigations and Follow-ups** * We fixed the bug. Media servers now handle forwarded data that arrives without the sender's identity instead of crashing. This fix was deployed to all regions by 08:49 UTC. * Raw data forwarded between media servers now includes the sender's identity, so receiving servers always know who sent it. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Read the full incident report →

Started Oct 28, 2026, 11:47 PM UTC· resolved Oct 29, 2026, 01:19 AM UTC

1. investigating Oct 28, 2026, 11:47 PM UTC

   We are currently investigating an increased rate of failing outbound calls in US East. We will post another update with more information as soon as possible.

2. investigating Oct 28, 2026, 11:51 PM UTC

   The LiveKit cloud dashboard is currently inaccessible. We are investigating this and will provide another update within 15 minutes.

3. investigating Oct 28, 2026, 11:57 PM UTC

   We believe there is also impact to inbound calls in the affected regions. We are working on identifying the cause and will update again within 15 minutes.

4. investigating Oct 29, 2026, 12:00 AM UTC

   We are currently attempting a mitigation. We will follow up with more information in the next 15 minutes.

5. investigating Oct 29, 2026, 12:09 AM UTC

   We are still determining the full scope of impact, but we believe that the Room Service has also been impacted. We will update again within 15 minutes.

6. monitoring Oct 29, 2026, 12:16 AM UTC

   We believe the mitigation was successful and are seeing services return to normal. The dashboard is accessible and error rates have gone down across services. We are continuing to monitor to make sure that the situation is resolved. We will update again as soon as possible.

7. identified Oct 29, 2026, 12:36 AM UTC

   While the US regions still appear to be stable, we are tracking degraded services including SIP, RTC, and APIs in our Mexico region. We will update again as soon as possible.

8. monitoring Oct 29, 2026, 12:41 AM UTC

   We've applied a mitigation for our Mexico region and services have returned to baseline. We are continuing to monitor and will update again within 15 minutes.

9. resolved Oct 29, 2026, 01:19 AM UTC

   This incident has been resolved. Between 23:30 and 23:58 UTC, a network disruption at an upstream infrastructure provider affected our US East region. During this window: Inbound SIP calls in US East failed at elevated rates, with affected calls rejected with 500 or 503 responses. Answered inbound calls were about 65% below normal across the window, and up to 90% below normal between 23:45 and 23:55 UTC. About 10% of outbound SIP call requests (CreateSIPParticipant) in US East failed, some returning 408 deadline_exceeded. Between 23:28 and 23:58 UTC, there were elevated rates of Room Service, Egress, and Ingress API request server errors in US East. Some participants saw failed room joins or increased packet loss. The LiveKit Cloud dashboard was inaccessible for part of this window. Network connectivity recovered at approximately 23:58 UTC, and error rates and call volumes returned to normal by 00:20 UTC. A related network issue also affected our much smaller Mexico region from 23:29 UTC, disconnecting active sessions and degrading service until traffic was moved to nearby regions at approximately 01:01 UTC. Sincere apologies for the impact. We will follow up with a postmortem.

10. postmortem Nov 01, 2026, 03:41 PM UTC

    ## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inbound calls were about 65% below normal across the window, and up to 90% below normal between 23:45 and 23:55. * **SIP outbound:** about 10% of `CreateSIPParticipant` requests in US East failed, some returning 408 `deadline_exceeded`. * **APIs:** significant server errors on Room Service, Egress, and Ingress APIs in US East. Recent versions of our server SDKs automatically retry failed API requests in other regions, which reduced the impact for customers on those versions. * **Cloud Agents:** agents in US East couldn't reach some of the model providers and other services they depend on, so agent sessions failed or degraded. Cloud Agents currently run in a single region, so they couldn't be quickly moved. * **Realtime:** some participants in US East saw failed room joins or increased packet loss. Sessions that were already connected kept working. * **Dashboard:** the LiveKit Cloud dashboard was inaccessible for part of the window. **Mexico \(23:29 to 01:01 UTC\)** Active sessions were disconnected and service was degraded until we moved traffic to nearby regions. ## Root Cause Starting at around 23:28 UTC, our cloud provider had a networking incident in its US East data center. It was a partial outage: our servers could still reach end users and our other regions, but lost access to our database and to certain external services, such as model providers. Anything that depended on those started failing. API requests and SIP call setup timed out, and agents couldn't reach the model providers they need to respond. Our automatic failover moves traffic away from a region when it becomes unhealthy. It didn't kick in here, for two reasons: 1. **Our health checks mostly look at our own software.** None of it failed. Every server in US East was up and responding. 2. **We could see that US East couldn't reach external services, but nothing acted on that automatically.** Those signals paged engineers, but they didn't feed into failover. As far as failover was concerned, US East looked healthy, so we kept sending traffic to a region that couldn't do much of the work. Cloud Agents had a separate gap. They currently run only in the data center specified in its deployment \(i.e. US East\) and we did not have a process to automatically migrate them quickly in case of a data center failure. Connectivity came back at around 23:58 UTC, once the provider mitigated the issue on their side. ## Timeline \(all times in UTC, starting October 28\) * **23:28**: Requests from US East to our database and certain external services start timing out. API errors follow shortly after. * **23:35**: Alerts fire for API errors in US East and an incident is opened. * **23:42**: Incident call starts. We initially focus on the API timeouts. * **23:45**: We remove an affected server to rule out a single bad instance. Failures continue. Since SIP, APIs, and agents are all affected, we start treating it as a regional network problem rather than a software bug. * **23:56**: Incident escalated. We start draining US East. * **23:58**: The provider's fix restores connectivity in US East. Impact in US East ends. * **00:36**: We identify degraded SIP, realtime, and API service in our Mexico region. * **01:01**: Mexico traffic moved to nearby regions, and service there recovers. ## Corrective Actions & Prevention * **Fail over based on external connectivity.** We're adding checks for whether a region can reach the services it depends on, and failover will act on them. If a region can't do its job, it should be drained automatically, even when every server in it looks healthy. * **Deploy Agents to multiple data centers.** We'll run agents in multiple data centers to have sufficient redundancy. * **Faster failover in agent dispatch.** If a server or an entire data center stops responding, agent dispatch will hand the job to another agent within seconds. * **Multi-cloud redundancy.** We're adding capacity with more than one cloud provider, so an outage at one provider can't take down a region.

Read the full incident report →

Started Oct 20, 2026, 11:09 PM UTC· resolved Oct 20, 2026, 11:50 PM UTC

1. investigating Oct 20, 2026, 11:09 PM UTC

   We are currently investigating higher than normal join latencies affecting a small percentage of participants in our US East and US Central regions. The latency increase appears to have begun at 21:30 UTC. Affected participants may notice increased join latencies higher than 5 seconds. The SDK retries automatically, and most of these joins succeed on retry. If your application stops waiting for an agent before then, those calls may time out. We will share another update as we learn more.

2. monitoring Oct 20, 2026, 11:25 PM UTC

   We identified connection drops at load balancers in our US Central and US East regions that caused a subset of connection latencies to take 5 seconds or longer. We applied a mitigation at 23:15 UTC, and we are seeing connection speeds return to normal. We are continuing to monitor before marking this resolved.

3. resolved Oct 20, 2026, 11:50 PM UTC

   Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem. We apologize for the impact and appreciate your patience.

4. postmortem Nov 05, 2026, 10:21 PM UTC

   **Root Cause** Between 2026-10-16 and 2026-10-23, some new connections to LiveKit Cloud in US Central, US East and EU Central got no response. The cause was a fault in our cloud provider's network load balancers. Some load balancers kept sending new connections to servers that were no longer in service. Most clients connected on a retry, so the main effect was a slower join. Some clients waited much longer. A routine server replacement and one of our recovery actions on October 20 also closed open connections from self-hosted agent workers. This caused agent dispatch failures for some customers. **Technical Details** Each LiveKit Cloud region has network load balancers from our cloud provider. A load balancer sends new connections only to servers that pass its health check. When servers are added or removed, the provider updates the health-check configuration of the load balancer. Our cloud provider found these causes: 1. A burst of health-check configuration updates overloaded the provider's health-check service. 2. Some of these updates failed. 3. A bug in the automatic recovery left the affected health checks inactive after the overload stopped. 4. As a result, the load balancers kept old health data. They continued to send new connections to servers that were no longer listening. Those connections got no response. The fault affected only some load balancers. Other load balancers with the same servers worked correctly. The load balancers reported no errors. **Agent dispatch failures.** Each self-hosted agent worker keeps one long-lived connection to LiveKit Cloud. The worker gets job offers on this connection. Two events on October 20 closed these connections: 1. At 16:14, a routine server replacement in US Central stopped one of our ingress servers. This closed about 485 agent worker connections at the same time. At that time, new traffic did not go to US Central, so these workers reconnected to US East. The workers of each affected customer were then split between two regions. Until about 17:40, agent dispatch failed or was slow for some customers. 2. At 23:15, we deleted the faulty load balancer in US Central. About 600 agent worker connections still used it. The workers reconnected within about 1 minute. But our servers did not detect that the old connections were closed. They continued to offer jobs on these connections until 23:32, when the connections timed out. Those job offers failed, and some jobs were not dispatched. **Timeline \(UTC\)** 2026-10-16 06:50 - Connection timeouts started in EU Central 2026-10-16 10:52 - We moved traffic away from the affected EU Central servers; the timeouts stopped. We opened a case with our cloud provider 2026-10-17 - We added probes that test each load balancer address from several regions 2026-10-20 08:00 - Slow agent joins started in US Central 2026-10-20 15:28 - We moved traffic away from US Central; join times went back to normal 2026-10-20 16:14 - A routine server replacement in US Central closed about 485 agent worker connections; agent dispatch failed or was slow for some customers until about 17:40 2026-10-20 21:30 - After traffic returned to US Central, slow joins started in US Central and US East 2026-10-20 22:30 - We moved traffic away from US Central again 2026-10-20 23:13 - We moved traffic away from US East 2026-10-20 23:15 - We deleted the faulty load balancer in US Central; about 600 agent worker connections still used it 2026-10-20 23:32 - The old agent worker connections timed out; agent dispatch went back to normal 2026-10-20 23:41 - We replaced the faulty load balancer in US Central and moved traffic back 2026-10-21 00:55 - We replaced the faulty load balancers in US East and moved traffic back 2026-10-21 03:13 - The fault occurred on another load balancer in US Central; we moved traffic away 2026-10-21 06:12 - We replaced that load balancer and moved traffic back to US Central 2026-10-21 07:02 - We replaced the faulty load balancers in EU Central and moved traffic back 2026-10-21 - Our cloud provider restored the correct health state on the 5 affected load balancers 2026-10-23 15:15 - The fault occurred at a lower rate on one load balancer in US East 2026-10-23 22:35 - Our probes detected it 2026-10-23 23:10 - We moved traffic off that load balancer; customer impact ended 2026-10-27 - Our cloud provider sent the root cause analysis **Scope of Impact** On each affected load balancer, 5% to 25% of new connections got no response. The load balancer fault did not affect connections that were already open. But a routine server replacement and one of our recovery actions on October 20 closed open connections from self-hosted agent workers \(see **Technical Details**\). * Oct 16 06:50–10:52 - Region: EU Central - About 5% of new connections timed out. This included connections for SIP calls. * Oct 20 08:00 – Oct 21 06:15 - Region: US Central - A share of new connections timed out. * Oct 20 16:14–17:40 - Region: US Central, US East - Some self-hosted agents did not get jobs or joined late. Some inbound calls were not answered. * Oct 20 21:30 – Oct 21 00:55 - Region: US East - 5% to 10% of new connections timed out. * Oct 20 23:16–23:32 - Region: US Central - Job offers to some self-hosted agent workers timed out. Some jobs were not dispatched. * Oct 23 15:15–18:15 and 20:45–23:10 - Region: US East - About 0.3–0.5% of new connections to the region timed out. The LiveKit SDKs stop a signal connection attempt after 5 seconds and try again. Most participants and agents joined on the retry, about 5 seconds later than usual. Clients with a longer connection timeout, or with no timeout, waited longer. Some self-hosted agents reported room joins that took 60–70 seconds. API calls to the affected regions could also time out or be slow. **Detection and Response** Customers reported the EU Central and US Central timeouts before our monitoring found them. Our checks tested each region as a whole, so a fault on one load balancer address was hidden. Most connections in a region still worked. On October 17, we added probes that test each load balancer address from several regions. These probes detected the October 23 fault. In each event, we moved traffic away from the affected region, replaced the faulty load balancers, and then moved traffic back. **Mitigations and Follow-ups** * Our cloud provider fixed the bug in a new release. With the fix, a failed health-check update is retried until it succeeds. The provider plans to start the deployment of this release in mid-November. We are working with the provider on when the fix reaches each region. * Each load balancer now sends connections only to the servers that run our ingress servers. Before, it included a larger group of servers, so each server that we added or removed caused a health-check update. Now the health checks change only when an ingress server moves. This change is live in most regions. We are completing it in the remaining regions, including US Central and US East. * Our probes now test each load balancer address and alert when the same connections fail again and again. * We changed how we replace a faulty load balancer. We now add a new load balancer, test it, and move traffic to it. The faulty load balancer stays in service until its open connections end, so agent workers on it are not cut off. * If an agent dispatch times out in one region, LiveKit Cloud now tries the next region. * We are working on a change that moves agent worker connections out of a region when we move traffic away from it. Then all agent workers of a customer reconnect to the same region. We sincerely apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Read the full incident report →

Started Oct 19, 2026, 05:02 PM UTC· resolved Oct 19, 2026, 05:34 PM UTC

1. investigating Oct 19, 2026, 05:02 PM UTC

   We are currently investigating reports of elevated API errors across our services in the Chicago region.

2. monitoring Oct 19, 2026, 05:11 PM UTC

   A fix has been implemented as of 17:00 UTC and we are seeing the errors return back to baseline. We are continuing to monitor before marking this resolved.

3. resolved Oct 19, 2026, 05:34 PM UTC

   Between 16:35-17:03 UTC, Chicago region experienced about 11% failure rates in API errors, failed agent dispatches, and issues connecting to rooms. We faced another wave of the same issue between 17:47-17:53 UTC where the failure rates were less than 10%. Room Service APIs called from other regions targeting to a room hosted in Chicago region were also affected during this period. API requests made with recent server SDK versions are automatically retried in other regions, so customers on those versions should not have seen these errors. The last related errors occurred at 17:53 UTC, and all affected services are operating normally since. We will share a detailed RCA with more information soon.

4. postmortem Oct 22, 2026, 12:47 AM UTC

   ## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Between 16:34 and 17:03 UTC, and again from 17:47 to 17:54 UTC, room management \(RoomService\), agent dispatch \(AgentDispatchService\) and SIP APIs failed for some rooms hosted in US Central. Because every region relies on the same coordination step, requests from other US regions that targeted those rooms failed as well. Affected requests returned a 503. Agent dispatch was the most affected. During the first window, 26.5% of rooms that requested an agent did not receive one. During the second window, 17.5% did not. Rooms hosted in other regions, and requests that did not target US Central rooms, were not affected. ## Root Cause Changes to room state, such as creating a room or dispatching an agent, must be confirmed by several servers before they take effect. A coordination service gathers those confirmations over our internal messaging system. If a quorum doesn't arrive within one second, the change fails. We recently rolled out a new overlay network for traffic between regions. It is more resilient to packet loss, but uses more CPU than its predecessor. In US Central, the message bus had little CPU headroom, and traffic was uneven: one of the three instances carried more than the other two combined, and its load had nearly doubled over the preceding hours. The overlay runs on every node and carries each message the message bus sends or receives. To guard against packet loss, it sends each message over several relay paths, so its CPU use climbs with message traffic. So that time-sensitive packet delivery is never throttled, the overlay can use more CPU than it requests, and it runs at a higher scheduling priority than the message bus. On the busiest instance's node, the two together pushed CPU above 90% and starved the message bus. At 16:34 UTC, routine node maintenance moved another messaging server to a new host, and its clients reconnected to the remaining servers. Shortly after, one coordination instance stopped getting replies through the overloaded server. New requests still reached it, but the responses it needed to finish them did not. Within four minutes it had stopped completing requests entirely. The coordination service is built to tolerate the loss of an instance: when one fails, another takes over its work. But this instance had not failed outright. Some messages still reached it, so it kept reporting itself healthy, and callers kept waiting on it instead of moving on. They retried, and request volume to it rose more than tenfold. Its unfinished updates also blocked changes to the same rooms in other US regions. We ended the first window by replacing the instance manually. The second window began when the instance that took over its busiest work stalled the same way, because the message bus was still short of CPU. Our monitoring did not catch the saturation. We monitored CPU for each pod against its own request, and relied on Kubernetes to keep nodes from being overloaded. That works only while every workload stays within its request. The message bus never exceeded its request, and the overlay's extra use did not show up in pod-level monitoring, so the node's saturation went unseen. Our error-rate alerts fired within four minutes, but nothing pointed to the node. ## Timeline \(UTC\) * 16:34 - Routine node maintenance moves a messaging server in US Central to a new host. API failures begin for rooms hosted in the region. * 16:38 - Automated monitoring detects elevated error rates across several US regions, and our team begins investigating. * 16:55 - The affected room-coordination instance is identified and replaced. * 17:03 - Error rates return to baseline. * 17:35 - We mark the incident resolved on this status page. The root cause is not yet known. * 17:47 - Errors return as the instance that took over the busiest work degrades in the same way. * 17:52 - That instance is replaced. Error rates return to baseline by 17:54. * 17:59 - A further instance showing early signs of the same problem is replaced. * 18:14 - New rooms are placed in other regions while the investigation continues. * 20:53 - CPU exhaustion on the messaging host is identified as the root cause. * 21:36 - Messaging servers in US Central are moved to larger hosts. * 22:51 - US Central resumes hosting new rooms. ## Corrective Actions & Prevention **Done** * **Increased CPU for the message bus in US Central.** The region's message bus now runs on larger nodes. * **Other regions resized.** We checked peak utilization across every region and moved messaging servers in regions at similar risk onto larger hosts. * **Automatic diagnostic capture.** The room-coordination service now records detailed diagnostics if a stall recurs. **Underway** * **Node-level CPU alerting.** We are adding CPU alerts for each node that carries internal traffic, alongside our pod-level monitoring, and alerts on CPU scheduling delays. * **Deadline-miss alerting.** A room-coordination instance that keeps missing its deadline will page our team. * **Automatic recovery for room coordination.** An instance that stops completing requests will hand off its work, and callers will route away from unhealthy instances. * **Retry limits.** Room-coordination requests will use backoff and a retry budget, so a stalled instance can no longer multiply request volume across regions. * **Messaging client reconnects.** Clients will reconnect automatically when a subscription stops receiving messages. * **Reserved CPU for the network overlay.** Hosts that run our messaging servers will reserve CPU for the overlay, so it cannot starve the servers it carries. We sincerely apologize for the disruption. If you have questions or were affected in ways not described here, please reach out to our support team.

Read the full incident report →

Started Oct 16, 2026, 04:04 PM UTC· resolved Oct 16, 2026, 05:30 PM UTC

1. investigating Oct 16, 2026, 04:04 PM UTC

   We are investigating degraded SIP performance in US West starting at 14:49 UTC affecting new outbound calls.

2. identified Oct 16, 2026, 04:12 PM UTC

   We have identified the source of the issue and are working to route around the impact.

3. monitoring Oct 16, 2026, 04:35 PM UTC

   We applied mitigation as of 15:58 UTC and SIP outbound connection success has returned to normal. We are continuing to monitor before marking this resolved.

4. monitoring Oct 16, 2026, 04:57 PM UTC

   We estimate that approximately 14% of outbound calls and 5% of inbound calls in US West failed between 14:49 UTC and 15:58 UTC. We will follow up as soon as possible with more details.

5. resolved Oct 16, 2026, 05:30 PM UTC

   This incident has been resolved. Between 14:49 UTC and 15:58 UTC, approximately 14% of outbound calls and 5% of inbound calls in US West failed. We will follow up with more details surrounding the root cause and preventative actions in a postmortem.

6. postmortem Oct 22, 2026, 02:22 PM UTC

   **Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affected SIP server started with an IPv6 address 2026-10-16 ~15:48 - Customers began reporting failures to our support team 2026-10-16 15:58 - SIP traffic began moving away from the final affected cluster due to automatic cluster draining 2026-10-16 15:59 - Last affected call; customer impact ended 2026-10-16 17:08 - Began rolling out the long term fix requiring the usage of IPv4 2026-10-16 ~21:28 - Fix rollout completed 2026-10-16 ~22:30 - SIP traffic restored to the affected cluster **Scope of Impact** Between 14:49 and 15:59 UTC on 2026-10-16, SIP calls handled by the two affected servers in US West failed. During that window, about 14% of outbound calls in US West were rejected at setup and about 5% of inbound calls were answered and then dropped without audio. **Detection and Response** Our customers noticed this before our monitoring did. Our alerts watch for error-level logs, but carrier rejections are logged as routine events since they happen during normal operation. We also weren't tracking call failure rates on individual servers so our on-call engineers were paged in manually about an hour after the first failure. **Mitigations and Follow-ups** * We deployed a fix so SIP servers only use IPv4 addresses when determining their public address. * We'll add safeguards and automated tests to ensure a SIP server's call setup messages only contain IPv4 addresses. * We are additionally adding improvements in monitoring so that we can proactively detect and prevent these sorts of failures. We apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Read the full incident report →

Started Oct 16, 2026, 10:55 AM UTC· resolved Oct 16, 2026, 11:59 AM UTC

1. investigating Oct 16, 2026, 10:55 AM UTC

   We are investigating intermittent timeouts when establishing new connections to our Frankfurt (EU) region. A small percentage of connection attempts may time out. We are working on a mitigation and will post an update as soon as possible.

2. monitoring Oct 16, 2026, 11:15 AM UTC

   We have applied a mitigation and new connections should no longer time out. We are monitoring to confirm that connection times have returned to normal.

3. resolved Oct 16, 2026, 11:59 AM UTC

   This incident is resolved. Connections to the Frankfurt region have returned to normal.

Read the full incident report →

Started Oct 13, 2026, 10:01 AM UTC· resolved Oct 13, 2026, 01:44 PM UTC

1. investigating Oct 13, 2026, 10:01 AM UTC

   We are investigating an issue with Agent Insights where agents are unable to upload their observability data at the end of their sessions. Some recent agent sessions may be missing from the dashboard's sessions list or shown as “In progress”. Agent sessions themselves are not affected. We will post an update shortly.

2. identified Oct 13, 2026, 10:45 AM UTC

   We have identified the cause of the Agent Insights issue and are working on a fix.

3. identified Oct 13, 2026, 11:40 AM UTC

   We are continuing to work on a fix for this issue.

4. identified Oct 13, 2026, 01:15 PM UTC

   Agent Insights uploads are now operational and session data has been storing normally since 12:40 UTC. We are still investigating an issue with Simulations. Simulation runs may complete without a transcript, summary, or verdict. We will post an update shortly.

5. resolved Oct 13, 2026, 01:44 PM UTC

   This incident has been resolved. Between 08:15 and 12:35 UTC, agents were not able to upload agent observability data to LiveKit's servers in all regions. Agent sessions completed normally, but, unfortunately, any Agent Insights data which attempted to upload within that window (traces/transcripts/logs/recordings) is unrecoverable. The ability to download existing Agent Insights data was impaired until 13:15 UTC. Agent Simulations were unavailable due to a upstream dependency and came back online at 13:25 UTC. Sincere apologies for the impact. We will follow up with a postmortem as soon as possible.

6. postmortem Oct 21, 2026, 11:40 PM UTC

   **Root Cause** During a security rotation of access credentials, we revoked a storage credential that we believed was no longer in use. It was still being used by our observability platform, which stores the data behind the dashboard and the Analytics API, the Agent Insights data that agents upload at the end of each session, and by the service that serves that data back for downloads and Agent Simulations. Once the credential was revoked, uploads failed in all regions. Restoring service took longer than a credential swap normally would for three reasons: * New credentials had to be set up, and each affected service had to be updated to use them. * The new credentials took longer than expected to become available in all regions \(about two hours in our US East region\). * Once the observability platform was working again, we found additional services that also needed updating. The incident was resolved once they were updated. **Timeline \(UTC\)** * 2026-10-13 08:13 - A storage credential is revoked during a security rotation. Observability uploads begin failing in all regions. * 08:15 - Agent Insights transcript and audio downloads and Agent Simulations begin failing. * 09:39 - Our team detects the failures and opens an incident. * 10:45 - The revoked credential is identified as the cause. * 11:16 - Affected services are reconfigured to use a new credential. * 12:40 - The new credential takes effect in the last region. Observability uploads recover in all regions. * 13:13 - Agent Insights downloads recover. * 13:26 - Agent Simulations recover. * 13:30 - Incident resolved. * 2026-10-17 - Recovery of observability data from the impact window completes \(see below\). **Scope of Impact** * **Observability uploads \(08:13 to 12:40 UTC, all regions\):** Observability data written during this window, including the data behind the dashboard and the Analytics API, failed to store. Nearly all agent sessions that ended in this window also failed to upload their Agent Insights data, so in Agent Insights these sessions were missing from the sessions list, or shown as "In progress". Agent sessions themselves ran normally, and media and calls were not affected. * **Data recovery:** In our resolved update we said all Agent Insights data from this window was unrecoverable. We have since restored most of it. **Session recordings and transcripts from this window are permanently lost.** All other observability data for these sessions has been restored and is visible in the dashboard and through the Analytics API. * **Agent Insights downloads \(08:15 to 13:13 UTC\):** Downloading transcripts and audio for existing sessions failed during the incident window. * **Agent Simulations \(08:15 to 13:26 UTC\):** Simulation runs couldn't load conversation transcripts, so they finished without a summary or verdict, or were cancelled. Runs from this window need to be re-run. * **Billing:** Agent Insights usage is only recorded after a successful upload, so failed uploads were **not** billed. **Detection and Response** We detected this issue about 85 minutes after it began. We did not have alerting on upload failures to observability storage, so it was caught when an engineer noticed elevated errors rather than by an automated alert. We also initially marked the incident resolved once uploads recovered, before confirming that Agent Simulations and downloads had recovered too. **Mitigations and Follow-ups** * We have added alerting that pages our on-call team when observability storage uploads fail, along with a continuous check that stored data can still be read and downloaded and an end-to-end test that verifies our storage credentials still work. * We have built tooling to replay and restore observability data from a past window, which we used to recover data from this incident. * We are moving these services to dedicated service credentials managed with our infrastructure and auditing other services for the same pattern. * We have updated our credential rotation process to verify every consumer of a credential before revoking it, and to account for regional propagation delays when rolling out replacements. * We are adding redundancy so a single failed credential can no longer stop observability uploads. We sincerely apologize for the disruption. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Read the full incident report →

Started Oct 06, 2026, 03:46 AM UTC· resolved Oct 06, 2026, 04:52 AM UTC

1. identified Oct 06, 2026, 03:46 AM UTC

   We are investigating an issue causing some cloud agent builds to be backed up in US East. This issue only impacts new builds for a small number of customers, existing agents are running without any impact.

2. resolved Oct 06, 2026, 04:52 AM UTC

   A fix was applied and all builds are back to normal.

Read the full incident report →

Started Oct 06, 2026, 12:10 AM UTC· resolved Oct 06, 2026, 02:01 AM UTC

1. investigating Oct 06, 2026, 12:10 AM UTC

   We are currently investigating reports of elevated errors affecting our RoomService API, beginning at 18:00 UTC. We will follow up with more details as soon as possible.

2. investigating Oct 06, 2026, 12:56 AM UTC

   We have determined that the elevated API errors are isolated to cross-region traffic between US East and US Central. We will provide further details shortly.

Read the full incident report →

Started Oct 05, 2026, 01:34 PM UTC· resolved Oct 05, 2026, 02:21 PM UTC

1. investigating Oct 05, 2026, 01:34 PM UTC

   We are investigating potential scaling issues with LiveKit Cloud hosted agents in eu-central resulting in increased join latencies. Existing agents are still functional and can serve sessions normally, but they may be slower to join new sessions. We are actively working on a mitigation to alleviate this issue. We will provide another update as soon as possible.

2. monitoring Oct 05, 2026, 01:40 PM UTC

   A fix has been implemented and we are monitoring to confirm full recovery.

3. resolved Oct 05, 2026, 02:21 PM UTC

   This incident has been resolved, and we have confirmed full recovery.

Read the full incident report →

Started Sep 15, 2026, 10:10 PM UTC· resolved Sep 15, 2026, 11:15 PM UTC

1. monitoring Sep 15, 2026, 10:10 PM UTC

   From 21:11 to 21:21 UTC we observed an occurrence of the same incident experienced earlier today impacting Global SIP, Ingress, and Egress due to the same underlying issue. We are currently operating normally and are continuing to monitor.

2. resolved Sep 15, 2026, 11:15 PM UTC

   We have resolved the underlying issue and have not observed any further occurrences of this incident. We will follow up with a detailed postmortem in the original incident.

3. postmortem Sep 17, 2026, 01:53 PM UTC

   Please see a full postmortem [here](https://status.livekit.io/incidents/dbqb4jhxcg9h).

Read the full incident report →

Started Sep 15, 2026, 05:04 PM UTC· resolved Sep 15, 2026, 06:16 PM UTC

1. investigating Sep 15, 2026, 05:04 PM UTC

   We are investigating reports of intermittently increased API latencies starting at 13:30 UTC. Users may observe increased latency spikes across all APIs.

2. monitoring Sep 15, 2026, 05:37 PM UTC

   We are still investigating, but believe that customer impact should be mostly mitigated - although some spikes may still be occurring. We are investigating reports of aborted egresses which we believe were also related to this issue. We will post another update as soon as possible.

3. monitoring Sep 15, 2026, 06:03 PM UTC

   We are continuing to monitor, but want to keep users up to date with how we believe impact may have materialized. The first impact window started at 13:40 UTC and impacted Ingress and Egress services for about 3 minutes. The second impact window started at 16:11 UTC and impacted SIP, Ingress, and Egress services for about 1 minute.

4. resolved Sep 15, 2026, 06:16 PM UTC

   We are resolving this incident as the underlying issue has been resolved. Thank you for your patience and apologies for the disruption. We will follow up with a postmortem as soon as possible.

5. postmortem Sep 17, 2026, 02:26 PM UTC

   ### Summary On 15 September, LiveKit Cloud experienced three periods of degraded performance between 13:34–13:41, 16:06–16:11 and 21:10–21:22 UTC. Ingress creation was the most affected: more than half of `CreateIngress` requests failed at the worst point. A number of in-progress recordings ended prematurely, and a small percentage of inbound SIP calls failed during a one-minute period \(during each impact window\). Realtime connections were not affected and media already flowing continued normally. The cause was a dropped connection to our primary metadata database that neither the client nor server side detected, which left a transaction open and holding row locks for up to 17 minutes. Requests queued behind those locks, and the sudden release of that backlog — not the wait itself — is what briefly degraded the wider platform. ### Root Cause Our services share a distributed global metadata database that stores ingress and egress state. At the time of the incident, a connection between one of our services and that database was dropped in a way neither end observed: the client treated the connection as closed and returned an error, while the database continued to consider the session live. Because the client had a transaction open at the time, the database kept that transaction's row locks held. The impact then came in two distinct phases. **While the lock was held**, only requests that needed the same rows were affected. Ingress creation failed, and recordings that could not report their status ended early. Most other APIs were unaffected, because they never touched the locked rows. **When the lock was released**, several hundred transactions that had queued behind it — some waiting more than seventeen minutes — all executed within a fraction of a second. That burst briefly saturated the database and degraded it for every service using it, not just those touching the original rows. This is why the broadest impact, including room management APIs and SIP participant creation, appears at the very end of each window rather than during it. The database only reclaimed the abandoned session when the operating system's TCP timeout expired, roughly 17 minutes after the connection was dropped. Two design choices amplified a single stuck row into regional impact: * **Recording workers treated "cannot report status" as "cannot accept work."** Every worker in a region reports through one shared service, so when that service became slow, all workers in the region stopped accepting new work at the same time and our router saw no available capacity. * **Several internal queries scanned the affected table without a narrowing filter.** That meant one locked row could block reads that were otherwise unrelated to it, which is what allowed the backlog to grow large enough to be disruptive on release. ### Scope of Impact Ingress creation was the most affected API: at the worst point in each window, 50.3%, 62.4% and 58.1% of `CreateIngress` requests returned server errors. Many failed requests are retried automatically, so the share of ingresses that ultimately could not be created is lower than those figures suggest. Starting a new egress was largely unaffected, staying under 2% throughout. SIP was affected at the end of each window, with `CreateSIPParticipant` reaching 7.50% and between 1.3% and 2.2% of attempted inbound calls failing during call setup; calls already connected were unaffected. Room management APIs stayed below 0.5%. Realtime connections were not affected, and media already flowing continued normally. The more consequential impact was to recordings already in progress. Up to 16.5% of egresses started during an affected window ended prematurely — 0.39% of all egresses that day — and did so without surfacing an error to indicate the recording had failed. ### Mitigations and Follow-ups Completed: * We have set a database-side idle transaction timeout so an abandoned transaction can no longer hold locks for more than 10 seconds. This caps both the wait and the size of any backlog that can accumulate behind it. We have verified this against a reproduction of the original failure. Underway: * We are removing an unnecessary transaction wrapper around single-statement writes, which shrinks the window in which a dropped connection can leave locks held. * We are adding database-side statement and lock timeouts so that no query can wait indefinitely on a lock. * We are changing recording workers so that a single status-reporting failure no longer removes an entire regional fleet from service. * We are making egress startup retry transient database errors rather than aborting the job. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Read the full incident report →

Started Sep 15, 2026, 12:08 PM UTC· resolved Sep 15, 2026, 03:30 PM UTC

1. investigating Sep 15, 2026, 12:08 PM UTC

   We are investigating an issue where users may be unexpectedly signed out of the LiveKit Cloud dashboard and unable to sign back in.

2. investigating Sep 15, 2026, 12:31 PM UTC

   We are continuing to investigate this issue affecting LiveKit Cloud dashboard, Analytic Services APIs and Billing APIs. Users may be unexpectedly signed out and unable to sign back in, and API requests may return 522 errors.

3. investigating Sep 15, 2026, 12:49 PM UTC

   We are continuing to investigate this issue.

4. identified Sep 15, 2026, 01:04 PM UTC

   We have identified the cause of the issue affecting the Cloud Dashboard and Analytics Services APIs, and recovery work is underway. Customers may continue to see errors accessing the Cloud Dashboard and Analytics Services APIs during this time. Room APIs, RTC APIs, and all other services remain healthy and unaffected. We have no indication of data loss from this incident. We will post another update as recovery progresses.

5. monitoring Sep 15, 2026, 02:29 PM UTC

   A fix has been deployed as of 14:23 UTC, and access to the Cloud Dashboard and Analytics Services APIs is returning to normal. We are continuing to monitor before marking this resolved. Recovery may take a few additional minutes for some clients as the change fully propagates; retrying or reconnecting will restore access.

6. resolved Sep 15, 2026, 03:30 PM UTC

   This incident has been resolved. Between 12:08 and 14:23 UTC, customers were unable to access the Cloud Dashboard and the Analytics Services APIs. The cause was a control plane failure in our last remaining legacy Kubernetes cluster, which took out a dependency of those services. Our user-facing services (RTC, SIP, and others) were unaffected, as they run redundantly across our modern clusters and have no dependency on the legacy cluster. We have no indication of data loss. Recovery took longer than we would like, as mitigation required working within the legacy cluster. Work was already underway to make the affected dependency redundant across our modern clusters, and that cluster is scheduled for decommissioning, which removes this class of failure. We will follow up with a postmortem.

Read the full incident report →

Started Sep 05, 2026, 11:36 PM UTC· resolved Sep 06, 2026, 01:12 AM UTC

1. identified Sep 05, 2026, 11:36 PM UTC

   We have identified an issue with LiveKit Cloud hosted agents in eu-central. Existing agents in eu-central are facing scaling issues resulting in increased join latencies. Existing agents are still functional and can serve sessions normally, but they may be slower to join new sessions. Additionally, new agents cannot be created in this region. We are actively working on a mitigation to alleviate this issue. We will provide another update as soon as possible.

2. monitoring Sep 06, 2026, 12:24 AM UTC

   We have applied a mitigation and believe that symptoms should be improving. We will update again with confirmation as soon as possible.

3. resolved Sep 06, 2026, 01:12 AM UTC

   The mitigation was successful and users shouldn't notice any issues with join latencies or creating new agents. We will follow up as soon as possible with a postmortem.

4. postmortem Sep 08, 2026, 09:59 PM UTC

   **Root Cause** A configuration change introduced a formatting error in the startup settings for the server pools that run hosted agents in our eu-central region. Newly provisioned servers failed to start, so the region could not add capacity. Agents that were already running were unaffected, but new agent deployments, version rollouts, and automatic scale-ups were delayed or stuck pending until the change was reverted. **Timeline \(UTC\)** `2026-09-04 23:56` - Configuration change deployed. The error only affected newly provisioned servers, so there was no immediate impact. `2026-09-05 18:07` - New servers began failing to start. Impact begins, intermittent at first. `2026-09-05 22:23` - Our monitoring alerted us to agent deployments stuck pending and we began investigating. `2026-09-06 00:50` - We identified the malformed startup configuration and reverted the change. `2026-09-06 01:17` - New capacity provisioned successfully, all pending deployments recovered, and we validated the fix. **Scope of Impact** Limited to hosted agents in our eu-central region. During the impact window, new agent deployments and rollouts were delayed or stuck pending, and automatic scale-up was blocked. Agents already running continued to serve sessions normally, and we estimate that, at the peak, roughly 1% of agent instances in the region were affected. No other regions or products were affected. **Mitigations and Follow-ups** * The faulty configuration change was reverted and provisioning has been stable since. * We are adding alerting for new servers that fail to start via this failure mode, which today fails silently. This lets us detect this class of failure directly and validate future changes to server provisioning quickly. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Read the full incident report →

Started Sep 04, 2026, 10:00 AM UTC· resolved Sep 04, 2026, 10:00 AM UTC

1. resolved Sep 04, 2026, 10:00 AM UTC

   A subset of room sessions that ended around 4 September 09:57 UTC were not recorded with an end time. These sessions continue to appear as active in the Cloud dashboard and in session records, and no `room_ended` end time is reflected for them. This was a reporting issue only. Rooms ended normally for participants, live traffic was not affected, and there is no impact on usage or billing. The underlying cause has been identified and fixed. Affected sessions from this window may continue to display as active; they can be safely disregarded.

Read the full incident report →

Started Sep 02, 2026, 09:03 PM UTC· resolved Sep 02, 2026, 09:45 PM UTC

1. investigating Sep 02, 2026, 09:03 PM UTC

   We are investigating elevated issues in calls connected to LiveKit Phone numbers where callee audio is not being heard by the caller. This is currently only experienced for numbers issues by LiveKit, and no other SIP provider connectivity should be affected. Our team is actively investigating this.

2. monitoring Sep 02, 2026, 09:25 PM UTC

   An upstream carrier reverted a codec change that was causing one-way audio issues on calls to LiveKit-issued phone numbers. As of 21:10 UTC calls are connecting normally and we are no longer seeing the issue. The upstream codec change has impacted calls from T-Mobile and AT&T carriers. No other SIP providers were affected. We are monitoring before marking this resolved.

3. resolved Sep 02, 2026, 09:45 PM UTC

   This incident has been resolved. We will follow up with a detailed postmortem as soon as possible.

Read the full incident report →

Started Sep 02, 2026, 08:48 AM UTC· resolved Sep 02, 2026, 12:03 PM UTC

1. monitoring Sep 02, 2026, 08:48 AM UTC

   Since ~06:30 UTC, outbound calls routed through Twilio are failing at an elevated rate. Twilio is reporting degraded voice connectivity (https://status.twilio.com/), affecting North America, Latin America, Europe, and the Middle East & Africa.

Read the full incident report →

Started Sep 02, 2026, 08:15 AM UTC· resolved Sep 02, 2026, 11:57 AM UTC

1. investigating Sep 02, 2026, 08:15 AM UTC

   We are currently investigating reports of our dashboard failing to load across multiple regions, beginning at 07:00 UTC. Other services remain unaffected and the impact is limited to the dashboard.

2. identified Sep 02, 2026, 09:01 AM UTC

   We have identified the cause: a problematic node in US-East and are deploying a fix. Impact continues to be limited to the dashboard.

3. monitoring Sep 02, 2026, 10:55 AM UTC

   A fix has been deployed as of 10:55 UTC, and the dashboard is now loading successfully. We are continuing to monitor before marking this resolved.

4. resolved Sep 02, 2026, 11:57 AM UTC

   This incident has been resolved and dashboard load times returned to normal by 10:55 UTC. As a note, our beta agent simulations feature was also impacted from approximately 06:07 - 07:52 UTC.

Read the full incident report →

Started Aug 26, 2026, 10:44 AM UTC· resolved Aug 26, 2026, 12:22 PM UTC

1. investigating Aug 26, 2026, 10:44 AM UTC

   We are investigating a period of elevated error rates and timeouts (08:47:00-08:53:35 UTC) for LiveKit Inference requests in our US East region. Service has since recovered and inference requests are completing normally. We are continuing to investigate the cause and confirm the full scope and duration of impact. A further update will follow.

2. resolved Aug 26, 2026, 12:22 PM UTC

   Between 08:47 and 08:54 UTC, the service that routes LLM, STT and TTS requests through LiveKit Inference in our US East region was unavailable, causing inference requests from agents in that region to fail or time out. Sessions depending on those responses may have degraded or ended prematurely. Replacement capacity came online at 08:54 UTC and request volumes returned fully to normal levels by 09:02 UTC. We have confirmed no further impact since then. The cause was a loss of capacity during automated infrastructure maintenance. We have identified fixes to prevent a recurrence.

Read the full incident report →

Started Aug 25, 2026, 10:33 AM UTC· resolved Aug 25, 2026, 11:47 AM UTC

1. investigating Aug 25, 2026, 10:33 AM UTC

   Between approximately 09:00 and 10:05 UTC, we observed intermittent elevated error rates on room and participant management APIs in the Mumbai region. Error rates have since returned to normal, and other regions were not affected. We are continuing to investigate the root cause and are monitoring the region.

Read the full incident report →

Started Aug 24, 2026, 10:11 AM UTC· resolved Aug 24, 2026, 02:14 PM UTC

1. monitoring Aug 24, 2026, 10:11 AM UTC

   We identified an issue where agent session analytics in the Cloud dashboard displayed zero concurrent sessions, beginning late on August 21 (UTC). This was a reporting issue only — agent sessions, calls, and all real-time services in these regions operated normally throughout. Ingestion has been restored and we are backfilling historical analytics for the affected window; some charts may show incomplete history until this completes. We are monitoring while the backfill finishes.

2. monitoring Aug 24, 2026, 01:55 PM UTC

   We are continuing to monitor while the historical data backfill completes, and will resolve this incident once all analytics for the affected window are fully restored.

3. resolved Aug 24, 2026, 02:14 PM UTC

   Agent session analytics in the Cloud dashboard have recovered and current data is reporting correctly for all affected projects. Some gaps may remain in historical data from the affected window (August 21–24 UTC); these will be backfilled.

Read the full incident report →

Started Aug 21, 2026, 09:18 AM UTC· resolved Aug 23, 2026, 09:00 PM UTC

1. resolved Aug 23, 2026, 09:00 PM UTC

   From 09:18 UTC on 21 August to 21:00 UTC on 23 August, 0.008% of egress recordings processed in our US East region were terminated before completion. Affected recordings were marked EGRESS_FAILED ("egress timed out"). For file outputs, media was not written to the configured destination; for HLS and streaming destinations, media was delivered up to the point of failure. The cause was node scale-down evicting egress workers with active recordings. A fix was deployed at 21:00 UTC on 23 August.

Read the full incident report →

Started Aug 20, 2026, 09:10 PM UTC· resolved Aug 20, 2026, 09:47 PM UTC

1. investigating Aug 20, 2026, 09:10 PM UTC

   We are currently investigating reports of elevated API errors across the Egress service. We will post another update in 5 minutes.

2. investigating Aug 20, 2026, 09:24 PM UTC

   We had a spike of errors between 20:52 and 21:01 UTC, and the errors have subsided to baseline as of 21:01 UTC. We are actively monitoring and will share more updates in 30 minutes.

3. resolved Aug 20, 2026, 09:47 PM UTC

   This incident has been resolved. Between 20:52 and 21:01 UTC, we observed an increase in failed egress and connector API requests. Errors have returned to baseline as of 21:01 UTC. We will follow up with a postmortem with further details.

4. postmortem Sep 10, 2026, 07:08 PM UTC

   On August 20, 2026, customers in our US East region experienced approximately 10 minutes of 503 errors on the LiveKit Ingress API and failures launching media track egress, between 20:52 and 21:01 UTC. All other regions were unaffected. ‌ **Root Cause** An internal routing service lost one of its two instances, and a concurrent infrastructure issue with our cloud provider was preventing new compute nodes from provisioning in US East, so the replacement couldn't start. At 20:51 UTC, an automated maintenance process removed that last instance, making the service fully unavailable. It recovered at 21:01 UTC once the evicted instance freed capacity for a replacement. ‌ **Timeline \(UTC\)** * Aug 20, 20:52 - Automated maintenance removes last instance in the pool; customer-facing errors begin * Aug 20, 21:01 - Service restored * Aug 21, 03:00 - Cloud provider resolves the underlying infrastructure issue; region fully healthy ‌ **Mitigations & Follow-ups** * The service is restored with two instances guaranteed to run on separate physical nodes. * We updated the automated maintenance process to avoid evicting the last healthy instance of a service. * We added disruption protection for the affected service and are continuing broader resilience improvements. * Our cloud provider resolved the infrastructure issue that had prevented new nodes from scaling up.

Read the full incident report →

Started Aug 19, 2026, 07:33 PM UTC· resolved Aug 20, 2026, 12:30 AM UTC

1. investigating Aug 19, 2026, 07:33 PM UTC

   We are currently investigating delayed session data ingestion on Cloud Dashboard. No services appear to be impacted and we don't expect any data to be lost.

2. identified Aug 19, 2026, 09:22 PM UTC

   Sessions data ingest remains delayed for about an hour. It's not impacting other dashboards. We've identified the source of the slow ingest and are working on a mitigation. We will update again in 30 mins.

3. identified Aug 19, 2026, 09:25 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Aug 19, 2026, 10:07 PM UTC

   The amount of data we are ingesting in the sessions view is outpacing our ability to ingest, thus causing a delay in ingestion. We are pursuing a few mitigations right now including moving to a larger database instance to relieve the processing bottleneck.

5. monitoring Aug 20, 2026, 12:01 AM UTC

   Mitigations have been deployed and we are now catching up to realtime.

6. resolved Aug 20, 2026, 12:30 AM UTC

   This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.

Read the full incident report →

Started Aug 17, 2026, 06:10 PM UTC· resolved Aug 17, 2026, 07:22 PM UTC

1. investigating Aug 17, 2026, 06:10 PM UTC

   We're investigating an elevated rate of failures when transferring active SIP calls in the US East region. SIP calls themselves remain connected, and inbound and outbound calling are otherwise operating normally.

2. investigating Aug 17, 2026, 06:22 PM UTC

   No transfer failures has been observed since 18:05 UTC and SIP transfers are currently completing normally. Impact was limited to a small percentage of calls between 15:48-16:08 UTC and 17:24-18:05 UTC. We're actively monitoring while we investigate the cause and put safeguards in place to prevent a recurrence.

3. resolved Aug 17, 2026, 07:22 PM UTC

   We have identified the root cause of both spikes and have ensured safeguards to prevent a recurrence. We've been monitoring since 18:05 UTC and have seen no further transfer failures, and have confirmed that SIP transfers are operating normally. We will follow up with a detailed post-mortem.

4. postmortem Sep 01, 2026, 11:44 AM UTC

   ## Summary On 17 August, between 17:24 and 18:05 UTC, a small percentage of SIP call transfers failed in our US East region during a routine configuration rollout. An issue in the automation that manages our SIP signaling servers prevented outgoing servers from being taken out of service safely, and those servers shut down while calls were still active on them. Transfers have completed normally since 18:05 UTC. Calls themselves stayed connected, and no other region was affected. ## Root Cause The configuration change restarts the servers that handle SIP signaling one at a time. Before a server is shut down, it is removed from service so that no new calls reach it, and it is then given some time to finish the calls it is already handling. In this case that removal did not complete expectedly and the outgoing servers kept receiving new calls for the entire drain duration, then shut down on schedule with calls still active on them. ## Timeline \(UTC\) * 16:52 - A routine configuration rollout begins in US East. * 17:24 - SIP transfer requests begin to fail for a subset of active calls. * 17:36 - Automated monitoring detects the elevated failure rate and our team begins investigating. * 17:40 - A further subset of active calls is affected. * 17:41 - Replacement capacity comes online. * 18:05 - Last of the impacted calls attempts a transfer and record a failure. ## Scope of Impact Only SIP call transfers \(`TransferSIPParticipant`\) in our US East region were affected, between 17:24 and 18:05 UTC. This represented 0.04% of all active calls in that window, and 1.1% of the calls that attempted a transfer. Customers who were impacted would have seen affected transfer requests return a 408; the underlying call stayed connected and only the transfer failed. Inbound and outbound calling were unaffected, as were calls that did not attempt a transfer, and no other region was affected. ## Mitigations and Follow-ups * We have deployed an alert for calls that end unexpectedly when a server shuts down. * We are changing our rollout process so that a server which cannot be removed from service safely halts the rollout. * We are preventing new calls from being routed to servers that are shutting down. * We are improving monitoring of the automation that manages SIP server rotation. * We are returning a more specific error when a transfer request cannot be delivered.

Read the full incident report →