LiveKit incident

2.5% of Inbound UDP Calls Dropping

Notice Resolved View vendor source →

LiveKit experienced a notice incident on September 10, 2025, lasting —. The incident has been resolved; the full update timeline is below.

Started
Sep 10, 2025, 05:54 PM UTC
Resolved
Aug 26, 2025, 05:00 AM UTC
Duration
—
Detected by Pingoru
Sep 10, 2025, 05:54 PM UTC

Update timeline

  1. resolved Sep 10, 2025, 05:54 PM UTC

    Our SIP load balancer runs inside of Kubernetes, which internally routes packets from the virtual pod IP to the host IP. We believe a linux kernel bug related to conntrack is causing certain packets (including our 200 OK responses) to be lost in that process. This bug impacts about 2.5% of inbound calls over UDP. Since we sent the 200, but it doesn't make it to the trunking provider, our system believes the call is connected when it isn't. Since TCP retries lost packets, it is not impacted by this bug. We will be rolling out a permanent fix in the next few weeks.