Is LiveKit down?

We are currently investigating intermittent 429 errors for Google Gemini models routed through LiveKit Inference.

We have identified the cause as an upstream limit on our Google Gemini API account. Automatic failover to an alternate Gemini deployment is in place and has reduced the impact, but a subset of requests to Google Gemini models may still intermittently return errors when failover capacity is exceeded. We are actively engaged with Google support to restore full capacity and will share further updates as we have them. Other models and providers remain unaffected.

This incident has been resolved. The errors were caused by an upstream issue at our model provider (Google) that incorrectly triggered an account-level usage cap on our Gemini API access, returning rate-limit errors for a subset of Gemini requests routed through LiveKit Inference. Google identified the cause, rolled back the change on their side, and raised our account limits to prevent recurrence. Gemini requests are now serving normally and error rates have returned to baseline. Other models and providers were unaffected throughout.

Between 13:00 and 15:05 UTC, a subset of requests to certain Google Gemini preview models (gemini-3.1-flash-lite and gemini-3-flash-preview) routed through LiveKit Inference returned errors due to a project-level misconfiguration. These preview models did not yet have model failover configured; all other models and providers were unaffected. This was resolved by routing the traffic via an alternate provider and fixing the misconfiguration. As resolution measures, we are extending model-level failover to cover these models and adding increased monitoring for such failures in the future.

We're investigating elevated latency affecting Ingress and SIP services in our Hyderabad region.

As of 09:21 UTC, we've applied a mitigation by routing traffic away from the Hyderabad region. Failure rates are returning to normal levels. We're continuing to monitor the issue.

We identified a connectivity problem in the Hyderabad cluster. As mitigation, we've drained the cluster to route traffic away from it. Mumbai traffic is unaffected and serving normally. We'll undrain Hyderabad once we're confident it's functioning normally.

Summary On June 2, 2026, a subset of users placing or receiving SIP calls through LiveKit Cloud's India region experienced call failures between 12:00 and 12:40 UTC. During a routine maintenance, both of our SIP-serving regions in India were temporarily disabled in error. With no SIP capacity available in India, traffic was routed to our Dubai region, where outbound calls to Indian numbers were rejected at the destination for region-pinned projects and inbound calls targeting India-specific endpoints failed to connect. We mitigated this by re-enabling SIP in the India region and restoring normal capacity. Timeline (UTC) 12:00 — Both India SIP regions disabled during maintenance; start of measurable customer impact 12:40 — India SIP regions re-enabled; customer impact ends Root cause The incident was caused by an operational error during planned maintenance. LiveKit operates three data centers in India, two of which serve SIP traffic. While disabling regions as part of our routine maintenance, the operator disabled both SIP-serving data centers, believing the third data center was also running SIP and would continue to carry customer traffic. Because the third data center does not serve SIP, this removed all SIP capacity in the India region and forced traffic to our Dubai region, where it could not complete region-pinned calls. Resolution Once we identified that India SIP capacity had been removed, we re-enabled SIP in the affected regions, which restored normal inbound and outbound calling in the region.

We're investigating elevated SIP call connection failures in our US Central region beginning at ~12:06 UTC. We are working to mitigate this issue.

We have routed the traffic away from the US Central region at 14:00 UTC and are seeing the connection failures returning to normal levels. We are continuing to monitor the issue.

We've identified the root cause as a single problematic media node in US Central, which has been suspended at 14:38 UTC. We are continuing to monitor before marking this resolved.

This incident has been resolved.

**Summary** On June 2, 2026, a small subset of users connected to LiveKit Cloud US Central region experienced elevated connection failures between 12:06 and 14:38 UTC. A single node in the region entered a degraded state in which it continued to accept new connections but could not reliably complete the real-time media connection those calls depend on. As a result, a portion of participant connections, including inbound and outbound SIP calls routed through Chicago failed to connect or dropped shortly after starting. We mitigated this by routing SIP traffic away from the affected server, suspending and removing the faulty node, and restoring normal service to the region. **Timeline \(UTC\)** * 12:06 — Start of measurable customer impact * 14:04 — Chicago SIP traffic drained as mitigation * 14:38 — Problematic server suspended; customer impact ends * 15:07 — Chicago SIP service fully restored **Root cause** The incident was caused by a "gray failure" of a single node in the Chicago region, a partial failure in which a server appears healthy to automated systems but is not actually functioning correctly. The server continued to be assigned new calls and reported itself as available, but could not reliably bring the underlying real-time media connections to a fully active state. **Resolution** We first drained SIP traffic out of the Chicago region to route the traffic to other regions, and once the failing node was identified, we suspended and removed the faulty node. After confirming error rates had returned to normal, we restored full SIP service to the region. **Corrective actions and prevention** We are actively addressing the two gaps that allowed a single failing node to impact the region: * **Error aversion in load balancing:** configuring our load balancing servers to automatically detect and steer traffic away from nodes exhibiting this class of failure. * **Monitoring:** closing the alerting gap so that "gray failure" servers which appear healthy but cannot complete connections are detected and alerted promptly.