Is LiveKit down?

We are currently investigating slow agent room joins affecting a small percentage of sessions in our US East and US Central regions. The rate increased beginning at 21:30 UTC. Affected agents take 5 seconds or longer to join a room instead of under 1 second. The SDK retries automatically, and most of these joins succeed on retry. If your application stops waiting for an agent before then, those calls may time out. We will share another update as we learn more.

We identified connection drops at load balancers in our US Central and US East regions that caused a subset of agent room joins to take 5 seconds or longer. We applied a mitigation at 23:15 UTC, and we are seeing connection speeds return to normal. We are continuing to monitor before marking this resolved.

Between 21:30 and 23:17 UTC, roughly 10% of all Real Time Communication(RTC) connections in our US East and US Central regions took 5 seconds or longer. We applied a mitigation at 23:15 UTC, and join times returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem.

We are currently investigating reports of elevated API errors across our services.

A fix has been implemented as of 17:00 UTC and we are seeing the errors return back to baseline. We are continuing to monitor before marking this resolved.

Between 16:35 and 17:03 UTC, Chicago region experienced about 11% failure rates in API errors, failed agent dispatches, and issues connecting to rooms. Room Service APIs called from other regions targeting to a room hosted in Chicago region were also affected during this period. The last related errors occurred at 17:02:42 UTC, and all affected services are operating normally since. We will share a detailed RCA with more information soon.

## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Between 16:34 and 17:03 UTC, and again from 17:47 to 17:54 UTC, room management \(RoomService\), agent dispatch \(AgentDispatchService\) and SIP APIs failed for some rooms hosted in US Central. Because every region relies on the same coordination step, requests from other US regions that targeted those rooms failed as well. Affected requests returned a 503. Agent dispatch was the most affected. During the first window, 26.5% of rooms that requested an agent did not receive one. During the second window, 17.5% did not. Rooms hosted in other regions, and requests that did not target US Central rooms, were not affected. ## Root Cause Changes to room state, such as creating a room or dispatching an agent, must be confirmed by several servers before they take effect. A coordination service gathers those confirmations over our internal messaging system. If a quorum doesn't arrive within one second, the change fails. We recently rolled out a new overlay network for traffic between regions. It is more resilient to packet loss, but uses more CPU than its predecessor. In US Central, the message bus had little CPU headroom, and traffic was uneven: one of the three instances carried more than the other two combined, and its load had nearly doubled over the preceding hours. The overlay runs on every node and carries each message the message bus sends or receives. To guard against packet loss, it sends each message over several relay paths, so its CPU use climbs with message traffic. So that time-sensitive packet delivery is never throttled, the overlay can use more CPU than it requests, and it runs at a higher scheduling priority than the message bus. On the busiest instance's node, the two together pushed CPU above 90% and starved the message bus. At 16:34 UTC, routine node maintenance moved another messaging server to a new host, and its clients reconnected to the remaining servers. Shortly after, one coordination instance stopped getting replies through the overloaded server. New requests still reached it, but the responses it needed to finish them did not. Within four minutes it had stopped completing requests entirely. The coordination service is built to tolerate the loss of an instance: when one fails, another takes over its work. But this instance had not failed outright. Some messages still reached it, so it kept reporting itself healthy, and callers kept waiting on it instead of moving on. They retried, and request volume to it rose more than tenfold. Its unfinished updates also blocked changes to the same rooms in other US regions. We ended the first window by replacing the instance manually. The second window began when the instance that took over its busiest work stalled the same way, because the message bus was still short of CPU. Our monitoring did not catch the saturation. We monitored CPU for each pod against its own request, and relied on Kubernetes to keep nodes from being overloaded. That works only while every workload stays within its request. The message bus never exceeded its request, and the overlay's extra use did not show up in pod-level monitoring, so the node's saturation went unseen. Our error-rate alerts fired within four minutes, but nothing pointed to the node. ## Timeline \(UTC\) * 16:34 - Routine node maintenance moves a messaging server in US Central to a new host. API failures begin for rooms hosted in the region. * 16:38 - Automated monitoring detects elevated error rates across several US regions, and our team begins investigating. * 16:55 - The affected room-coordination instance is identified and replaced. * 17:03 - Error rates return to baseline. * 17:35 - We mark the incident resolved on this status page. The root cause is not yet known. * 17:47 - Errors return as the instance that took over the busiest work degrades in the same way. * 17:52 - That instance is replaced. Error rates return to baseline by 17:54. * 17:59 - A further instance showing early signs of the same problem is replaced. * 18:14 - New rooms are placed in other regions while the investigation continues. * 20:53 - CPU exhaustion on the messaging host is identified as the root cause. * 21:36 - Messaging servers in US Central are moved to larger hosts. * 22:51 - US Central resumes hosting new rooms. ## Corrective Actions & Prevention **Done** * **Increased CPU for the message bus in US Central.** The region's message bus now runs on larger nodes. * **Other regions resized.** We checked peak utilization across every region and moved messaging servers in regions at similar risk onto larger hosts. * **Automatic diagnostic capture.** The room-coordination service now records detailed diagnostics if a stall recurs. **Underway** * **Node-level CPU alerting.** We are adding CPU alerts for each node that carries internal traffic, alongside our pod-level monitoring, and alerts on CPU scheduling delays. * **Deadline-miss alerting.** A room-coordination instance that keeps missing its deadline will page our team. * **Automatic recovery for room coordination.** An instance that stops completing requests will hand off its work, and callers will route away from unhealthy instances. * **Retry limits.** Room-coordination requests will use backoff and a retry budget, so a stalled instance can no longer multiply request volume across regions. * **Messaging client reconnects.** Clients will reconnect automatically when a subscription stops receiving messages. * **Reserved CPU for the network overlay.** Hosts that run our messaging servers will reserve CPU for the overlay, so it cannot starve the servers it carries. We sincerely apologize for the disruption. If you have questions or were affected in ways not described here, please reach out to our support team.

We are currently investigating degraded SIP performance in US West.

We have identified the source of the issue and are working to route around the impact.

We appllied mitigation as of 15:58 UTC and SIP outbound connection success has returned to normal. We are continuing to monitor before marking this resolved.

We estimate that approximately 13% of outbound calls in US West failed between 14:49 UTC and 15:58 UTC. We will follow up as soon as possible with more details.

This incident has been resolved. Between 14:49 UTC and 15:58 UTC, approximately 14% of outbound calls and 5% of inbound calls in US West failed. We will follow up with more details surrounding the root cause and preventative actions in a postmortem.

**Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affected SIP server started with an IPv6 address 2026-10-16 ~15:48 - Customers began reporting failures to our support team 2026-10-16 15:58 - SIP traffic began moving away from the final affected cluster due to automatic cluster draining 2026-10-16 15:59 - Last affected call; customer impact ended 2026-10-16 17:08 - Began rolling out the long term fix requiring the usage of IPv4 2026-10-16 ~21:28 - Fix rollout completed 2026-10-16 ~22:30 - SIP traffic restored to the affected cluster **Scope of Impact** Between 14:49 and 15:59 UTC on 2026-10-16, SIP calls handled by the two affected servers in US West failed. During that window, about 14% of outbound calls in US West were rejected at setup and about 5% of inbound calls were answered and then dropped without audio. **Detection and Response** Our customers noticed this before our monitoring did. Our alerts watch for error-level logs, but carrier rejections are logged as routine events since they happen during normal operation. We also weren't tracking call failure rates on individual servers so our on-call engineers were paged in manually about an hour after the first failure. **Mitigations and Follow-ups** * We deployed a fix so SIP servers only use IPv4 addresses when determining their public address. * We'll add safeguards and automated tests to ensure a SIP server's call setup messages only contain IPv4 addresses. * We are additionally adding improvements in monitoring so that we can proactively detect and prevent these sorts of failures. We apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

We are investigating intermittent timeouts when establishing new connections to our Frankfurt (EU) region. A small percentage of connection attempts may time out. We are routing traffic away from the affected infrastructure and will post an update once this is complete.

We have routed traffic away from the affected infrastructure in our Frankfurt (EU) region, and new connections should no longer time out. We are monitoring to confirm that connection times have returned to normal.

This incident is resolved. Connections to the Frankfurt region have returned to normal.

We are investigating an issue with Agent Insights. Some recent agent sessions may be missing from the Sessions list or shown as “In progress”. Agent sessions themselves are not affected. We will post an update shortly.

We have identified the cause of the agent insights issue and are working on a fix.

We are continuing to work on a fix for this issue.

Agent Insights is operational, and session data has been storing normally since 12:40 UTC. We are still investigating an issue with Simulations. Simulation runs may complete without a transcript, summary, or verdict. We will post an update shortly.

This incident has been resolved. Between 08:15 and 12:35 UTC, agents were not able to upload agent observability data to LiveKit's servers in all regions. Agent sessions completed normally, but, unfortunately, any Agent Insights data which attempted to upload within that window (traces/transcripts/logs/recordings) is unrecoverable. The ability to download existing Agent Insights data was impaired until 13:15 UTC. Agent Simulations were unavailable due to a upstream dependency and came back online at 13:25 UTC. Sincere apologies for the impact. We will follow up with a postmortem as soon as possible.

**Root Cause** During a security rotation of access credentials, we revoked a storage credential that we believed was no longer in use. It was still being used by our observability platform, which stores the data behind the dashboard and the Analytics API, the Agent Insights data that agents upload at the end of each session, and by the service that serves that data back for downloads and Agent Simulations. Once the credential was revoked, uploads failed in all regions. Restoring service took longer than a credential swap normally would for three reasons: * New credentials had to be set up, and each affected service had to be updated to use them. * The new credentials took longer than expected to become available in all regions \(about two hours in our US East region\). * Once the observability platform was working again, we found additional services that also needed updating. The incident was resolved once they were updated. **Timeline \(UTC\)** * 2026-10-13 08:13 - A storage credential is revoked during a security rotation. Observability uploads begin failing in all regions. * 08:15 - Agent Insights transcript and audio downloads and Agent Simulations begin failing. * 09:39 - Our team detects the failures and opens an incident. * 10:45 - The revoked credential is identified as the cause. * 11:16 - Affected services are reconfigured to use a new credential. * 12:40 - The new credential takes effect in the last region. Observability uploads recover in all regions. * 13:13 - Agent Insights downloads recover. * 13:26 - Agent Simulations recover. * 13:30 - Incident resolved. * 2026-10-17 - Recovery of observability data from the impact window completes \(see below\). **Scope of Impact** * **Observability uploads \(08:13 to 12:40 UTC, all regions\):** Observability data written during this window, including the data behind the dashboard and the Analytics API, failed to store. Nearly all agent sessions that ended in this window also failed to upload their Agent Insights data, so in Agent Insights these sessions were missing from the sessions list, or shown as "In progress". Agent sessions themselves ran normally, and media and calls were not affected. * **Data recovery:** In our resolved update we said all Agent Insights data from this window was unrecoverable. We have since restored most of it. **Session recordings and transcripts from this window are permanently lost.** All other observability data for these sessions has been restored and is visible in the dashboard and through the Analytics API. * **Agent Insights downloads \(08:15 to 13:13 UTC\):** Downloading transcripts and audio for existing sessions failed during the incident window. * **Agent Simulations \(08:15 to 13:26 UTC\):** Simulation runs couldn't load conversation transcripts, so they finished without a summary or verdict, or were cancelled. Runs from this window need to be re-run. * **Billing:** Agent Insights usage is only recorded after a successful upload, so failed uploads were **not** billed. **Detection and Response** We detected this issue about 85 minutes after it began. We did not have alerting on upload failures to observability storage, so it was caught when an engineer noticed elevated errors rather than by an automated alert. We also initially marked the incident resolved once uploads recovered, before confirming that Agent Simulations and downloads had recovered too. **Mitigations and Follow-ups** * We have added alerting that pages our on-call team when observability storage uploads fail, along with a continuous check that stored data can still be read and downloaded and an end-to-end test that verifies our storage credentials still work. * We have built tooling to replay and restore observability data from a past window, which we used to recover data from this incident. * We are moving these services to dedicated service credentials managed with our infrastructure and auditing other services for the same pattern. * We have updated our credential rotation process to verify every consumer of a credential before revoking it, and to account for regional propagation delays when rolling out replacements. * We are adding redundancy so a single failed credential can no longer stop observability uploads. We sincerely apologize for the disruption. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.