LiveKit Outage History

We have fixed the data pipeline delay and all sessions should be loading now in the Cloud Dashboard.

An internal one-off analysis job ran some expensive DB queries which slowed down load times of the dashboard's sessions page. The job was removed and load times are now back to baseline.

Between approximately 10:30 and 11:55 UTC on July 28, customers with telephony workloads in our London region experienced elevated error rates on SIP APIs. Small number of call transfers were affected. We mitigated by removing the affected infrastructure from service at 11:48 UTC, and error rates returned to normal by 11:55 UTC.

On July 27, our San Jose region experienced a minor networking incident beginning at 16:14 UTC. Users connected to this region may have experienced reconnects and increased session start times. Traffic was routed away from this region at 16:39 UTC and errors have since returned to baseline.

On July 23, between 07:50 and 08:07 UTC, media servers in our London region experienced a host-level network fault, causing them to fail health checks and restart. Users connected to these servers may have experienced dropped connections, or interrupted egress recordings. The affected servers were removed and replaced. The region has been operating normally since.

Error rates for room creation in our London region have remained normal since 09:06 am UTC and we are no longer seeing any issues.

Connection failures in our Japan region have returned to baseline as of 18:43 UTC.

As of 18:45 UTC, all features of Cloud Dashboard are fully restored and functional.

Service is fully restored as of 17:18 UTC. Traffic is being successfully routed to other regions.

This incident has been resolved. The errors were caused by an upstream issue at our model provider (Google) that incorrectly triggered an account-level usage cap on our Gemini API access, returning rate-limit errors for a subset of Gemini requests routed through LiveKit Inference. Google identified the cause, rolled back the change on their side, and raised our account limits to prevent recurrence. Gemini requests are now serving normally and error rates have returned to baseline. Other models and providers were unaffected throughout.

Between 13:00 and 15:05 UTC, a subset of requests to certain Google Gemini preview models (gemini-3.1-flash-lite and gemini-3-flash-preview) routed through LiveKit Inference returned errors due to a project-level misconfiguration. These preview models did not yet have model failover configured; all other models and providers were unaffected. This was resolved by routing the traffic via an alternate provider and fixing the misconfiguration. As resolution measures, we are extending model-level failover to cover these models and adding increased monitoring for such failures in the future.

On 2026-06-11 at 13:00 UTC, a serialization bug was introduced via frontend components which rendered some services unavailable via the dashboard for some users. These services included SIP trunk and dispatch rule creation/management as well as several advanced features in Agent Builder. A fix was pushed on 2026-06-17 at 17:43 UTC which resolved the issue.

We identified a connectivity problem in the Hyderabad cluster. As mitigation, we've drained the cluster to route traffic away from it. Mumbai traffic is unaffected and serving normally. We'll undrain Hyderabad once we're confident it's functioning normally.

All missing data has now been backfilled and the incident is resolved.

We have been monitoring the clusters for the past 16 hours and have not found any further disruptions.

Between ~17:16 and 17:21 UTC on May 7, 2026, inter-region connectivity between our US East region and other regions was briefly degraded due to a network connectivity issue in our underlying cloud provider's data center. During this window, sessions that relied on cross-region track relay through US East may have experienced brief connection failures or reconnections for some participants. Our system recovered automatically once the cloud provider's data center network connectivity was restored.