LiveKit Outage History

Started Aug 12, 2026, 06:34 PM UTC· resolved Aug 12, 2026, 08:09 PM UTC

1. investigating Aug 12, 2026, 06:34 PM UTC

   We are currently investigating alerts for elevated server error rates on LiveKit RoomService APIs (GetParticipant, ListParticipants, RemoveParticipant) across multiple regions. The impact appears to be intermittent and limited to a subset of room-management API requests. We will follow with more details as soon possible.

2. monitoring Aug 12, 2026, 07:20 PM UTC

   Error rates on ListParticipants API requests returned to normal as of 18:58 UTC. We will follow up with more details as soon possible.

3. resolved Aug 12, 2026, 08:09 PM UTC

   Error rates on ListParticipants API requests recovered at 18:58 UTC and have remained stable since.

Read the full incident report →

Started Aug 07, 2026, 09:22 PM UTC· resolved Aug 07, 2026, 10:27 PM UTC

1. investigating Aug 07, 2026, 09:22 PM UTC

   We are currently investigating automated alerting which triggered for degraded Room service performance in the US West and US Central regions beginning around 19:45 UTC. We are working to determine if there is any customer-facing impact. If there is impact, we don't currently have reason to believe that it is widespread. We will update again within 30 minutes.

2. monitoring Aug 07, 2026, 09:57 PM UTC

   We have applied a mitigation to the affected regions and are now monitoring to ensure that performance signals return to baseline. We are still working on determining impact (including how users can determine if they were affected) and will share those details as soon as possible.

3. resolved Aug 07, 2026, 10:27 PM UTC

   This incident has been resolved. Between 19:45 and 20:00 UTC, US West and US Central experienced intermittent degradation in Room Service and SIP performance. We rerouted traffic and performance has returned to baseline as of 22:00 UTC. We will follow up with a postmortem, which will include further details regarding scale and scope.

Read the full incident report →

Started Aug 06, 2026, 07:47 PM UTC· resolved Aug 07, 2026, 02:51 AM UTC

1. identified Aug 06, 2026, 07:47 PM UTC

   Between approximately 18:45 and 19:15 UTC today, a subset of LiveKit Inference requests using the google/gemma-4-31b-it model returned errors. Affected agent sessions would have seen an LLM request error on those requests. Other models and other LiveKit services were not affected. No action is needed on your part. If you continue to see errors, please reach out to support. We apologize for the disruption.

2. identified Aug 06, 2026, 07:51 PM UTC

   We are actively investigating elevated error rate on LiveKit Inference (google/gemma-4-31b-it)

3. monitoring Aug 06, 2026, 09:08 PM UTC

   A burst of requests has increased the failure rate. Both primary and secondary model providers failed to fulfill the requests at the time; we are root-causing the issue. We are continuing to monitor.

4. resolved Aug 07, 2026, 02:51 AM UTC

   Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very near future we will be onboarding additional providers to avoid these kinds of issues.

Read the full incident report →

Started Aug 04, 2026, 08:45 PM UTC· resolved Aug 04, 2026, 10:11 PM UTC

1. monitoring Aug 04, 2026, 08:45 PM UTC

   We are monitoring elevated timeouts that affected LiveKit Inference between 14:05 and approximately 15:00 UTC today. During that window, roughly 3% of LLM requests using the google/gemma-4-31b-it model timed out. Impacted customers would have seen LLM request timeouts in their agent sessions (for example, APITimeoutError in agent logs). Other models were not affected. We have identified the cause. During a period of increased traffic, response times for this model slowed, and a defect in our health-monitoring logic incorrectly marked a backup deployment as unhealthy and removed it from rotation, preventing requests from failing over as designed. Error rates returned to baseline by approximately 15:00 UTC, and a fix for the health-monitoring defect is in progress. We are continuing to monitor before marking this resolved. No action is needed on your part. If you continue to see timeouts, please reach out to support. We apologize for the disruption.

2. resolved Aug 04, 2026, 10:11 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 03, 2026, 03:30 PM UTC· resolved Aug 03, 2026, 07:48 PM UTC

1. investigating Aug 03, 2026, 03:30 PM UTC

   We are currently investigating an issue where the sessions view does not populate in the Cloud Dashboard. Real-time connectivity is not affected and no data loss is observed.

2. investigating Aug 03, 2026, 04:51 PM UTC

   We have traced the underlying issue to a backlog in our data pipeline, and we are continuing to investigate the root cause while working to clear the backlog. This data ingest delay results in the Sessions view returning no results for any time range ending at the current time, including all Quick Ranges options in the dashboard. As a temporary workaround, select "Custom range" and set the end time at least 30 minutes in the past to load your sessions. Real-time connectivity is not affected and no data has been lost.

3. resolved Aug 03, 2026, 07:48 PM UTC

   We have fixed the data pipeline delay and all sessions should be loading now in the Cloud Dashboard.

Read the full incident report →

Started Jul 30, 2026, 10:55 PM UTC· resolved Jul 30, 2026, 10:55 PM UTC

1. resolved Jul 30, 2026, 10:55 PM UTC

   Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period.

2. postmortem Jul 30, 2026, 10:56 PM UTC

   ## Summary Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period. ## Root Cause A bug in an internal configuration service caused webhook settings to be dropped from a project's cached configuration whenever that project's record was updated. Once triggered, our system stopped delivering webhooks until the cache refreshed from the database, up to one hour later. Because of this caching behavior, the impact was intermittent: webhooks would stop, then resume on their own, then stop again after the next project update. Additionally, the "test webhook" feature in the dashboard reads webhook configuration through a different path and was unaffected, so test webhooks succeeded even while real event delivery was failing. ## Timeline \(UTC\) * 2026-07-16 23:48 - Change containing the bug reached production; intermittent impact begins * 2026-07-24 - Some customer reports of missing webhooks received; investigation begins * 2026-07-28 16:23 - Root cause identified; fix developed and rollout begins * 2026-07-29 16:26 - Fix deployed to all regions; incident resolved ## Scope of Impact Customers using webhooks were affected across all regions between 2026-07-16 and 2026-07-29, with most impact occurring after 2026-07-22. A project was affected only if its configuration was updated during the window, and each occurrence lasted up to one hour. Affected projects saw `room_started` and `participant_*` webhooks silently not delivered, while `room_finished` webhooks and dashboard webhook tests often continued to work. If you observed missing webhooks and had recently made any settings changes to your project, you were likely impacted. ## Mitigations and Follow-ups * The bug has been fixed and a regression test has been added. * We added logging of webhook delivery response codes to improve visibility into delivery failures. * We are adding per-project webhook delivery monitoring and alerting that can detect missing deliveries independent of overall volume. * We are unifying the configuration paths used by webhook delivery and the dashboard webhook test, so a passing test reflects actual delivery behavior. * We are making the configuration cache lifetime adjustable so similar issues can be mitigated quickly in the future.

Read the full incident report →

Started Jul 28, 2026, 10:30 AM UTC· resolved Jul 28, 2026, 11:55 AM UTC

1. resolved Jul 28, 2026, 02:46 PM UTC

   Between approximately 10:30 and 11:55 UTC on July 28, customers with telephony workloads in our London region experienced elevated error rates on SIP APIs. Small number of call transfers were affected. We mitigated by removing the affected infrastructure from service at 11:48 UTC, and error rates returned to normal by 11:55 UTC.

Read the full incident report →

Started Jul 27, 2026, 04:14 PM UTC· resolved Jul 27, 2026, 04:39 PM UTC

1. resolved Jul 27, 2026, 05:25 PM UTC

   On July 27, our San Jose region experienced a minor networking incident beginning at 16:14 UTC. Users connected to this region may have experienced reconnects and increased session start times. Traffic was routed away from this region at 16:39 UTC and errors have since returned to baseline.

Read the full incident report →

Started Jul 23, 2026, 07:50 AM UTC· resolved Jul 23, 2026, 08:07 AM UTC

1. resolved Jul 24, 2026, 04:21 PM UTC

   On July 23, between 07:50 and 08:07 UTC, media servers in our London region experienced a host-level network fault, causing them to fail health checks and restart. Users connected to these servers may have experienced dropped connections, or interrupted egress recordings. The affected servers were removed and replaced. The region has been operating normally since.

Read the full incident report →

Started Jul 14, 2026, 09:49 AM UTC· resolved Jul 14, 2026, 10:41 AM UTC

1. monitoring Jul 14, 2026, 09:49 AM UTC

   Between 08:54 am and 09:06 am UTC, some requests to create new rooms in our London region failed with errors. Error rates returned to normal by 09:06 am UTC and a fix has been applied. We are continuing to monitor.

2. resolved Jul 14, 2026, 10:41 AM UTC

   Error rates for room creation in our London region have remained normal since 09:06 am UTC and we are no longer seeing any issues.

Read the full incident report →

Started Jul 06, 2026, 06:40 PM UTC· resolved Jul 07, 2026, 10:30 AM UTC

1. investigating Jul 06, 2026, 06:40 PM UTC

   We are currently investigating increased connection failures affecting our Japan region beginning at 18:18 UTC.

2. resolved Jul 06, 2026, 06:54 PM UTC

   Connection failures in our Japan region have returned to baseline as of 18:43 UTC.

Read the full incident report →

Started Jun 22, 2026, 01:05 PM UTC· resolved Jun 22, 2026, 02:40 PM UTC

1. monitoring Jun 22, 2026, 01:05 PM UTC

   We have begun routing traffic away from US Central to mitigate reports of WebRTC connection failures and SIP transfer errors.

2. monitoring Jun 22, 2026, 01:59 PM UTC

   Update: As soon as the US Central region became unresponsive around 12:25 UTC, traffic was automatically re-routed to the nearest healthy region. Customers may have noticed failed API requests for approximately 1 minute while the re-route took place. Failed SIP Transfers continued until we finished manually draining the impacted region at 13:09 UTC. We are also investigating impact to Agent Dispatches and will post another update when we know more.

3. resolved Jun 22, 2026, 02:40 PM UTC

   We are resolving this incident as the mitigation is in place and successful. We have received some reports of self-hosted agents which connect to US Central needing to be restarted in order to continue receiving dispatches and will investigate opportunities to improve this behavior. Apologies for the disruptions and we will follow up with a postmortem as soon as possible.

4. postmortem Jun 23, 2026, 08:32 AM UTC

   ## Summary LiveKit Cloud is a globally distributed system. Our data centers continuously probe one another, and our overlay network uses those probes to choose healthy paths between regions. When a brief network disruption occurs in or between data centers, the overlay is designed to mark affected paths as unhealthy, route traffic around them, and return them to "healthy" once the underlying network has cleared. On June 18, 2026 \(US East 1\) and again on June 22, 2026 \(US Central\), a short packet-loss event in the affected data center triggered a compound failure in this mechanism. Two separate defects, one in the overlay network and one in our link-monitoring service, interacted in a way that left the affected region effectively isolated from other regions long after the underlying network had cleared. For most customers and most products, the impact was bounded. Our global routing automatically moved traffic to other US regions within approximately one minute of onset. SIP transfers anchored in the affected region took longer to recover but cleared within approximately thirty minutes. A small number of LiveKit Agents running self-hosted workers experienced an extended outage in agent dispatch, up to about two hours in some cases. The majority of agent workloads continued to receive jobs normally after the routing flip. The workaround at the time was to restart the affected agent worker process so that it re-registered against a controller in a healthy region. This is a separate defect in our agent dispatch routing, and we are fixing it. This post-mortem describes what happened, why the second incident occurred so soon after the first, and the steps we are taking to prevent recurrence. ## Impact * **Realtime connections:** minimal impact. Clients automatically reconnected to alternative data centers. * **API:** approximately 1 minute of degraded availability while traffic rerouted to a nearby region. * **SIP transfers:** approximately 30 minutes for in-progress transfers anchored in the affected region. * **Agent dispatch:** approximately 1 minute for the majority of agents. A small number of agent workers saw extended dispatch outages of up to approximately 2 hours until the worker process was restarted. ## Root Cause The trigger in both incidents was the same: a short packet-loss event in the affected data center. The expected behavior was that affected paths would be marked unhealthy briefly while the overlay rerouted traffic around them, and would return to "healthy" once the underlying network cleared. Two separate defects interacted in a way that kept routing state stuck long after the network had recovered: 1. **Inbound and outbound freshness collapsed onto a single indicator.** In the overlay network, the two directions of a connection \(inbound and outbound\) shared a single "freshness" indicator. When fresh data arrived in one direction, our code incorrectly assumed data in the other direction was also fresh. As a result, the affected region continued to treat its outbound connections to other regions as unreachable based on stale inbound data, even after outbound traffic was not impacted. 2. **Link-monitor key exchange could de-sync during a disruption.** Our link-monitoring service relies on a key exchange between regions to validate probe traffic. The packet-loss event caused this key exchange to fall temporarily out of sync between the affected region and its neighbors. With probes failing validation, the link monitors in neighboring regions marked the affected region as "to avoid," and held that state past the actual network recovery. Either defect on its own would not have produced an outage. Together, they produced a stuck state in both directions: neighboring regions believed they could not reach the affected region \(because probe validation kept failing\), and the affected region believed it could not reach the others \(because the freshness collapse kept its outbound state pinned to "unreachable"\). Traffic stayed routed away from the affected region until the state was manually cleared. Following Incident 1 on June 18, we developed and merged a fix for the overlay freshness defect on the same day. The fix shipped as a new version of the overlay software and was being validated in staging at the time of Incident 2. On June 22, the same compound failure occurred on US Central, which was still running the unpatched version of the overlay. At the time of writing, the fix is now in active rollout to production. ## Corrective Actions & Prevention **Immediate \(in active rollout\)** * **Roll the overlay freshness fix to production region-by-region.** As of this writing, the rollout is underway and we expect global production coverage in the next two days. We are giving this rollout priority given the severity. **Underway** * **Improve resilience of the link-monitor probing process to key-exchange de-sync.** This addresses the second of the two defects described above. Eliminating it is necessary to prevent the compound failure even after the overlay freshness fix is in place. _ETA: Thursday, June 25._ * **Migrate existing agent worker connections on health changes.** We correctly detected the data center as unhealthy and rerouted new traffic away quickly, but did not migrate existing agent worker connections. The change is to automatically migrate agent worker connections to healthy data centers as soon as the health indicator starts failing. _ETA: June 30._ ## For customers running self-hosted agent workers Until our agent dispatch routing fix is deployed, the most reliable mitigation if you observe a dispatch outage is to restart your agent worker processes. This causes them to re-register against controllers in healthy regions. To catch the condition early and automate the recovery, consider adding a health check that: * Tracks the time since the worker last received a dispatch \(or last completed a session\). * Triggers a restart of the worker process if this exceeds an expected idle window for your workload. We recognize that this workaround places a burden on customers and is not a substitute for the underlying server-side fix. Eliminating it is a priority. We sincerely apologize for the disruption to customers whose traffic and agent workloads were affected. Thank you for your patience, and we welcome any additional feedback from customers who were impacted.

Read the full incident report →

Started Jun 18, 2026, 05:49 PM UTC· resolved Jun 18, 2026, 07:10 PM UTC

1. investigating Jun 18, 2026, 05:49 PM UTC

   We are investigating reports of the Sessions and Agent Insights pages not being accessible on the LiveKit Cloud Dashboard. There is no data loss or ongoing session impact associated with this.

2. investigating Jun 18, 2026, 06:39 PM UTC

   Session data (except Agent Insights) on the dashboard can still be accessed by changing the timeline on the Sessions page to "Past 7 days" or greater. We're still working to restore full access.

3. resolved Jun 18, 2026, 07:10 PM UTC

   As of 18:45 UTC, all features of Cloud Dashboard are fully restored and functional.

Read the full incident report →

Started Jun 18, 2026, 05:09 PM UTC· resolved Jun 18, 2026, 05:40 PM UTC

1. investigating Jun 18, 2026, 05:09 PM UTC

   We are currently investigating connection failure reports in US East.

2. monitoring Jun 18, 2026, 05:32 PM UTC

   We have routed traffic away from US East region and connection failure errors are back to baseline. We are continuing to investigate and monitor SIP transfer errors.

3. resolved Jun 18, 2026, 05:40 PM UTC

   Service is fully restored as of 17:18 UTC. Traffic is being successfully routed to other regions.

Read the full incident report →

Started Jun 16, 2026, 09:49 AM UTC· resolved Jun 16, 2026, 08:42 PM UTC

1. monitoring Jun 16, 2026, 09:49 AM UTC

   We identified an issue in the India region where a small number of active sessions may have ended earlier than expected. Affected sessions could experience egress recordings stopping mid-session. We've deployed a mitigation and are monitoring to confirm resolution. Other regions are unaffected.

2. resolved Jun 16, 2026, 08:42 PM UTC

   We have not seen further instances of this issue since the mitigation was applied. We will follow up with a postmortem as soon as possible.

3. postmortem Jun 23, 2026, 05:46 PM UTC

   **Summary** Between June 5 and June 16, 2026, an intermittent loss of connectivity between internal systems caused room state to be altered incorrectly in a single region in India. As a result, a small number of active sessions in that region ended earlier than expected, and affected egress recordings could stop mid-session. While the intermittent issue was undetected for some time, the impacted number of egresses was low and limited to the one affected region in India - other regions were unaffected. **Timeline \(UTC\)** * 2026-06-05 05:00 - Intermittent loss of connectivity between internal systems begins causing room state to be incorrectly altered in one India region, leading to some egresses being terminated prematurely. * 2026-06-16 06:45 - Mitigation was deployed and monitoring began. Eligible connections were moved to a different region and added redundancy was deployed to the affected internal tooling. * 2026-06-16 20:42 - No further instances were observed since the mitigation was applied. Incident was marked resolved. **Resolution** We mitigated the outage by moving eligible connections to a different region and by adding redundancy to the affected internal tooling so it could better tolerate intermittent connectivity loss. As a longer-term fix, we hardened the communication channels between the involved systems. We have not observed any further occurrences since the mitigation was applied, and we are continuing to evaluate additional improvements to our tolerance for this class of failure.

Read the full incident report →

Started Jun 11, 2026, 05:52 PM UTC· resolved Jun 11, 2026, 11:47 PM UTC

1. investigating Jun 11, 2026, 05:52 PM UTC

   We are currently investigating intermittent 429 errors for Google Gemini models routed through LiveKit Inference.

2. identified Jun 11, 2026, 07:09 PM UTC

   We have identified the cause as an upstream limit on our Google Gemini API account. Automatic failover to an alternate Gemini deployment is in place and has reduced the impact, but a subset of requests to Google Gemini models may still intermittently return errors when failover capacity is exceeded. We are actively engaged with Google support to restore full capacity and will share further updates as we have them. Other models and providers remain unaffected.

3. resolved Jun 11, 2026, 11:47 PM UTC

   This incident has been resolved. The errors were caused by an upstream issue at our model provider (Google) that incorrectly triggered an account-level usage cap on our Gemini API access, returning rate-limit errors for a subset of Gemini requests routed through LiveKit Inference. Google identified the cause, rolled back the change on their side, and raised our account limits to prevent recurrence. Gemini requests are now serving normally and error rates have returned to baseline. Other models and providers were unaffected throughout.

Read the full incident report →

Started Jun 11, 2026, 01:00 PM UTC· resolved Jun 11, 2026, 05:43 PM UTC

1. resolved Jun 17, 2026, 09:53 PM UTC

   On 2026-06-11 at 13:00 UTC, a serialization bug was introduced via frontend components which rendered some services unavailable via the dashboard for some users. These services included SIP trunk and dispatch rule creation/management as well as several advanced features in Agent Builder. A fix was pushed on 2026-06-17 at 17:43 UTC which resolved the issue.

Read the full incident report →

Started Jun 11, 2026, 01:00 PM UTC· resolved Jun 11, 2026, 03:05 PM UTC

1. resolved Jun 11, 2026, 03:50 PM UTC

   Between 13:00 and 15:05 UTC, a subset of requests to certain Google Gemini preview models (gemini-3.1-flash-lite and gemini-3-flash-preview) routed through LiveKit Inference returned errors due to a project-level misconfiguration. These preview models did not yet have model failover configured; all other models and providers were unaffected. This was resolved by routing the traffic via an alternate provider and fixing the misconfiguration. As resolution measures, we are extending model-level failover to cover these models and adding increased monitoring for such failures in the future.

Read the full incident report →

Started Jun 05, 2026, 09:06 AM UTC· resolved Jun 05, 2026, 11:08 AM UTC

1. investigating Jun 05, 2026, 09:06 AM UTC

   We're investigating elevated latency affecting Ingress and SIP services in our Hyderabad region.

2. investigating Jun 05, 2026, 10:05 AM UTC

   As of 09:21 UTC, we've applied a mitigation by routing traffic away from the Hyderabad region. Failure rates are returning to normal levels. We're continuing to monitor the issue.

3. resolved Jun 05, 2026, 11:08 AM UTC

   We identified a connectivity problem in the Hyderabad cluster. As mitigation, we've drained the cluster to route traffic away from it. Mumbai traffic is unaffected and serving normally. We'll undrain Hyderabad once we're confident it's functioning normally.

Read the full incident report →

Started Jun 03, 2026, 12:00 PM UTC· resolved Jun 03, 2026, 12:40 PM UTC

1. resolved Jun 04, 2026, 11:45 PM UTC

   Summary On June 2, 2026, a subset of users placing or receiving SIP calls through LiveKit Cloud's India region experienced call failures between 12:00 and 12:40 UTC. During a routine maintenance, both of our SIP-serving regions in India were temporarily disabled in error. With no SIP capacity available in India, traffic was routed to our Dubai region, where outbound calls to Indian numbers were rejected at the destination for region-pinned projects and inbound calls targeting India-specific endpoints failed to connect. We mitigated this by re-enabling SIP in the India region and restoring normal capacity. Timeline (UTC) 12:00 — Both India SIP regions disabled during maintenance; start of measurable customer impact 12:40 — India SIP regions re-enabled; customer impact ends Root cause The incident was caused by an operational error during planned maintenance. LiveKit operates three data centers in India, two of which serve SIP traffic. While disabling regions as part of our routine maintenance, the operator disabled both SIP-serving data centers, believing the third data center was also running SIP and would continue to carry customer traffic. Because the third data center does not serve SIP, this removed all SIP capacity in the India region and forced traffic to our Dubai region, where it could not complete region-pinned calls. Resolution Once we identified that India SIP capacity had been removed, we re-enabled SIP in the affected regions, which restored normal inbound and outbound calling in the region.

Read the full incident report →

Started Jun 02, 2026, 02:05 PM UTC· resolved Jun 02, 2026, 03:18 PM UTC

1. investigating Jun 02, 2026, 02:05 PM UTC

   We're investigating elevated SIP call connection failures in our US Central region beginning at ~12:06 UTC. We are working to mitigate this issue.

2. monitoring Jun 02, 2026, 02:17 PM UTC

   We have routed the traffic away from the US Central region at 14:00 UTC and are seeing the connection failures returning to normal levels. We are continuing to monitor the issue.

3. monitoring Jun 02, 2026, 02:58 PM UTC

   We've identified the root cause as a single problematic media node in US Central, which has been suspended at 14:38 UTC. We are continuing to monitor before marking this resolved.

4. resolved Jun 02, 2026, 03:18 PM UTC

   This incident has been resolved.

5. postmortem Jun 04, 2026, 02:01 PM UTC

   **Summary** On June 2, 2026, a small subset of users connected to LiveKit Cloud US Central region experienced elevated connection failures between 12:06 and 14:38 UTC. A single node in the region entered a degraded state in which it continued to accept new connections but could not reliably complete the real-time media connection those calls depend on. As a result, a portion of participant connections, including inbound and outbound SIP calls routed through Chicago failed to connect or dropped shortly after starting. We mitigated this by routing SIP traffic away from the affected server, suspending and removing the faulty node, and restoring normal service to the region. **Timeline \(UTC\)** * 12:06 — Start of measurable customer impact * 14:04 — Chicago SIP traffic drained as mitigation * 14:38 — Problematic server suspended; customer impact ends * 15:07 — Chicago SIP service fully restored **Root cause** The incident was caused by a "gray failure" of a single node in the Chicago region, a partial failure in which a server appears healthy to automated systems but is not actually functioning correctly. The server continued to be assigned new calls and reported itself as available, but could not reliably bring the underlying real-time media connections to a fully active state. **Resolution** We first drained SIP traffic out of the Chicago region to route the traffic to other regions, and once the failing node was identified, we suspended and removed the faulty node. After confirming error rates had returned to normal, we restored full SIP service to the region. **Corrective actions and prevention** We are actively addressing the two gaps that allowed a single failing node to impact the region: * **Error aversion in load balancing:** configuring our load balancing servers to automatically detect and steer traffic away from nodes exhibiting this class of failure. * **Monitoring:** closing the alerting gap so that "gray failure" servers which appear healthy but cannot complete connections are detected and alerted promptly.

Read the full incident report →

Started May 28, 2026, 02:53 PM UTC· resolved May 28, 2026, 08:13 PM UTC

1. investigating May 28, 2026, 02:53 PM UTC

   We are currently investigating reports of spikes in participant connection latency and errors

2. identified May 28, 2026, 03:35 PM UTC

   We are observing database connection timeouts in the US East region and are currently seeing an impact to all services in the region. We are currently draining the region and routing away all traffic to other regions.

3. monitoring May 28, 2026, 03:50 PM UTC

   US East drain is complete and all new traffic is now routing to other regions. Error rate is trending down and we'll continue monitoring.

4. monitoring May 28, 2026, 04:27 PM UTC

   Connection errors have fully cleared since the US East drain completed at 15:44 UTC. We'll continue monitoring. We will be sharing a detailed RCA.

5. monitoring May 28, 2026, 06:27 PM UTC

   Service is operating normally with traffic routing through other regions. We are working on bringing US East back online and will share a detailed RCA once complete.

6. resolved May 28, 2026, 08:13 PM UTC

   Connection errors have remained cleared since the US East drain at 15:44 UTC, and as of 19:30 UTC, US East is back online. We'll share a detailed RCA in the coming days.

7. postmortem May 30, 2026, 06:18 PM UTC

   ## Summary LiveKit Cloud validates incoming API and connection requests via an internal authentication service, accessed over gRPC. The service has been in place since 2022 and is designed for resilience: it runs as multiple redundant instances, has pod-level health monitoring that automatically reaps unhealthy pods, and is backed by in-memory caches so it can survive transient database failures. On 2026-05-28, between 13:55 UTC and 15:45 UTC, a percentage of requests and new connections in our US East region failed or timed out. The root cause was a rare failure mode on a single instance of the authentication service. The instance remained reachable and its TCP connections stayed alive, but it began responding to gRPC requests extremely slowly, in a way that did not trip our existing pod-level health checks or cause gRPC clients to fail over. We sincerely apologize to customers whose traffic was disrupted. We've let you down, and we are taking this very seriously. In addition to the corrective actions outlined below, we are performing a thorough audit of our systems for other failure modes we may not have anticipated. ## Timeline and Impact \(all times in UTC on 2026-05-28\) * **13:55**: Elevated rate of authentication failures observed. * **14:15**: On-call team notified; investigation begins. * **14:46**: Error rate continues to climb; incident escalated to Sev 1. * **15:35**: US East drained; error rate begins to subside. * **16:32**: Root cause identified; faulty instance shut down. * **16:32**: Webhooks backlogged during the incident begin to deliver. Between 2026-05-28 13:55 UTC and 15:35 UTC, a percentage of API requests and new connection attempts in our US East region failed or timed out. This included new participant connections, SIP connection requests, and agent connections originating in US East. The blast radius of this incident was substantially reduced by our local in-process auth cache. Each service maintains an in-memory cache of recently-seen authentication details, which allowed the majority of requests to continue flowing whenever they landed on a server that had already cached the relevant credentials. The failures concentrated on requests that landed on servers without those credentials cached, which disproportionately impacted customers with lower overall traffic to US East. Existing realtime sessions in US East continued to operate, and all traffic in other regions was unaffected. The impact subsided once US East was drained and traffic was rerouted to neighboring regions. There was also a secondary effect for customers who subscribe to outgoing webhooks. During the incident, webhook deliveries that depended on the affected auth path were queued internally while waiting for a valid signing token. Once the faulty instance was removed and the previously stuck operations unblocked, the queued webhooks were delivered in a burst rather than smoothly over time. Customers whose webhook endpoints have limited throughput may have observed a short-lived spike that exceeded their normal load. ## Root Cause The trigger was a rare hardware failure mode on one instance of our authentication service. Typically, when hardware fails, the underlying machine is shut down either by the hypervisor or by Kubernetes \(via failed health checks\), and gRPC clients reconnect to a healthy instance. In this case, the faulty machine did not terminate. It remained reachable and continued to respond, but extremely slowly. The TCP connection to that instance stayed up, so gRPC clients that had pinned their requests onto that connection continued to use it, and pod-level health checks did not fire. As a result, a percentage of services in US East were unable to fail over to healthy instances and saw their auth requests time out. Two additional fallback mechanisms did not engage as fully as we had designed: 1. **Client-driven cross-region failover.** LiveKit clients can automatically fail over from one region to a neighboring healthy region. However, the failover path itself depended on passing the same authentication gate, so it failed for the same reason. 2. **Local in-process auth cache refresh path.** As noted in the Impact section, the in-process cache absorbed a significant share of requests during the incident. However, services always attempt to refresh auth details against the auth service first to ensure they have the most recent data, and fall back to the cache only when that refresh fails or times out. The refresh attempt has a 5-second timeout. Many client SDKs use a default request timeout shorter than 5 seconds and gave up before the cache fallback could engage, even when the relevant credentials were present in the cache. When the impact was first observed, our initial signal pointed to database connection timeouts in US East. Because of that, we did not immediately drain the region. Two reasons informed that decision: 1. only a portion of requests were failing \(some connections were still succeeding\) 2. if the underlying database had truly been the cause, draining US East would have shifted that load onto neighboring regions and risked spreading the impact rather than containing it. After confirming that the database was healthy and that the issue was isolated to US East, we drained the region. Error rates began dropping immediately as traffic moved to neighboring regions. ## Corrective Actions & Prevention This incident exposed a machine failure mode we had not yet designed for. The following changes have been implemented and will be rolled out in the next week: * **Detect and time out slow gRPC connections from internal clients.** Internal gRPC clients will detect connections that are alive but unresponsive, and tear them down so they can re-establish against a healthy instance. * **Improved load balancing between gRPC servers.** Retries from internal clients will spread across alternative instances, so a single slow server cannot black-hole a meaningful share of traffic. * **Reduce inline auth refresh timeout to 1 second.** This allows the local in-process auth cache fallback to engage well within the timeout budget that client SDKs typically use. * **Allow client-driven cross-region failover to function when auth has failed.** Region failover will no longer depend on passing the auth gate, so a degraded auth service in one region cannot prevent clients from reaching a healthy region. We sincerely apologize for the disruption to customers whose traffic was affected. Thank you for your patience, and we welcome any additional feedback from customers who were impacted.

Read the full incident report →

Started May 26, 2026, 07:22 PM UTC· resolved May 26, 2026, 09:32 PM UTC

1. investigating May 26, 2026, 07:22 PM UTC

   We are investigating reports of Agent Observability not accessible for some sessions.

2. identified May 26, 2026, 07:53 PM UTC

   We have identified an issue causing partial data unavailability in the LiveKit Cloud Dashboard: For some users, Agent Observability is temporarily unavailable for some sessions. Some sessions in the 24-hour filter are showing as Active even though they have ended. A fix has been identified and we are backfilling the missing data.

3. monitoring May 26, 2026, 08:13 PM UTC

   A fix has been implemented and we are seeing no more data outage for newly created sessions. We are monitoring the fix and are currently working on backfilling the missing data.

4. resolved May 26, 2026, 09:32 PM UTC

   All missing data has now been backfilled and the incident is resolved.

Read the full incident report →

Started May 22, 2026, 08:26 PM UTC· resolved May 23, 2026, 04:36 PM UTC

1. investigating May 22, 2026, 08:26 PM UTC

   We have observed network instability impacting cross-region traffic from Frankfurt beginning at 18:15 UTC. Traffic has been redirected from Frankfurt as of 19:55 UTC. Participants originally connecting via Frankfurt may see slightly increased latency until the region is restored.

2. monitoring May 23, 2026, 12:17 AM UTC

   We have re-enabled our Frankfurt cluster and are monitoring for further disruptions.

3. resolved May 23, 2026, 04:36 PM UTC

   We have been monitoring the clusters for the past 16 hours and have not found any further disruptions.

Read the full incident report →

Started May 15, 2026, 04:57 PM UTC· resolved May 16, 2026, 02:50 AM UTC

1. investigating May 15, 2026, 04:57 PM UTC

   We are investigating intermittent connectivity disruptions affecting a subset of sessions, caused by internal network degradation between several of our US clusters (most heavily impacting US East). Affected sessions may see brief (<5 min) interruptions to media relay. We are actively investigating. Further updates to follow.

2. monitoring May 15, 2026, 09:16 PM UTC

   We have not observed further instances of connectivity disruptions since 13:19 UTC. We are continuing to investigate why our reroute mechanism did not activate in these isolated instances and are monitoring for further disruptions.

3. resolved May 16, 2026, 02:50 AM UTC

   We are resolving this incident as we have not observed any further connectivity disruptions. We will follow up with a postmortem once available.

4. postmortem May 22, 2026, 07:36 AM UTC

   ## Summary LiveKit Cloud runs as a distributed realtime network, with data centers around the world interconnected via dedicated networking. Even with dedicated fiber, momentary disruptions between any two data centers can and do occur. To handle these blips, we have built a comprehensive set of resilience mechanisms that automatically reroute and relay traffic over alternate healthy paths. For example, if data centers A and B cannot reach one another cleanly but both can reach data center C, we use C as a relay so that traffic flows A to C to B. Under normal circumstances, network blips between our data centers are handled transparently and customers do not notice them. Between 2026-05-07 and 2026-05-19, a small number of these otherwise routine network blips did become customer-visible in our US regions. Affected sessions experienced a brief interruption to media \(under 5 minutes\) before recovering on a new path. Realtime connections, SIP calls, agent processes, and the wider fleet were not otherwise impacted. ## Impact Five short windows of disruption were observed, each tied to a brief network blip between certain clusters: * 2026-05-12 17:16 UTC * 2026-05-12 05:00 UTC * 2026-05-12 17:00 UTC * 2026-05-13 19:00 UTC * 2026-05-15 07:30 UTC * 2026-05-19 15:01 UTC The majority of sessions traversed alternate paths normally and were not affected. A subset of sessions whose traffic happened to be relayed through a region in the specific failure state described below experienced a media interruption of up to ~5 minutes before re-routing onto a healthy path. ## Root Cause On 2026-05-11, a change was deployed that altered the relay process. The change introduced a subtle bug that required two conditions to occur simultaneously to manifest: 1. The relay region was under-utilized and had not cached a particular piece of state needed by the relay process. 2. A large volume of traffic was directed to that relay region in a very short window of time. When both conditions were present, the relay process would be stuck and would take minutes to fully catch up. During that time, neither endpoint of the relayed session could continue to receive media from the other. Because both conditions are narrow \(a cold-cache relay region absorbing a sudden burst of traffic\), the bug did not surface during pre-deploy testing, and it did not trigger on every network blip. It only manifested when a real network disruption happened to redirect a sufficiently large burst of traffic to a relay region that had not warmed its cache. Once that occurred on a given relay, sessions flowing through it stalled until traffic shifted off that path. We tracked down the root cause on 2026-05-20, and a fix has been fully rolled out across the fleet. ## Corrective Actions & Prevention * **Improved integration tests for relay locks under load.** We are adding tests that exercise the relay process with cold caches and sudden traffic bursts, so this class of contention is caught before deploy. We are also auditing related lock paths to ensure they behave correctly under similar load profiles. We sincerely apologize for the disruption to customers whose sessions were affected during these windows. Thank you for your patience, and we welcome any additional feedback from customers who were impacted.

Read the full incident report →