LiveKit experienced a minor incident on December 31, 2025 affecting Global SIP, lasting 8h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 31, 2025, 07:24 PM UTC
Less than 5 percent of inbound calls are seeing delays and no response back to INVITEs. We are actively investigating this.
- monitoring Dec 31, 2025, 09:00 PM UTC
A code change has been pushed to mitigate the issue. We are currently monitoring
- resolved Jan 01, 2026, 04:02 AM UTC
We've added more stats to track if the issue is recurring and we have not seen this issue recur. We are working on the postmortem with timelines and will update it soon.
- postmortem Jan 03, 2026, 07:01 PM UTC
**Root Cause:** The issue was caused by an incorrectly formed header in BYE requests, which triggered an infinite forwarding loop of SIP messages on our SIP load balancing service. This resulted in the server repeatedly forwarding messages to itself, causing resource exhaustion on the affected machines. During these periods, processing of incoming INVITE requests were delayed on affected servers. **Time window of impact:** We saw the first instances of these delays happen at 2025-12-28 19:00:00 UTC. While we pushed the first patches to mitigate this issue to majorly impacted regions on 2026-01-01 04:00:00 UTC, the fix was rolled out globally on 2026-01-02 19:00:00 UTC. **Scope of impact:** The incident had a much lower impact vs the 5% assessment we made during the incident based on limited data. The incident is now categorized as degraded service as opposed to a partial outage. In terms of actual impact to calls, out of the small percentage of calls that rang longer \(stats below\), a smaller subset of them could have timed out depending on what the timeout settings of the inbound calling trunk were. Regions affected: Eastern US Daily impact \(percentage of calls that rang longer than 2s - all times in UTC\): 1. 2025-12-28: 0.076% 2. 2025-12-29: 0.061% 3. 2025-12-30: 0.103% 4. 2025-12-31: 0.0046% Worst impacted hourly windows \(percentage of calls that rang longer than 2s - all times in UTC\): 1. 2025-12-28 19:00 \+ 1h: 1.01% 2. 2025-12-29 18:00 \+ 1h: 0.32% 3. 2025-12-30 16:00 \+ 1h: 0.37% 4. 2025-12-31 19:00 \+ 1h: 0.38% **Mitigations:** * We already applied a quick fix to mitigate the issue over the past few days. * We will be rolling out code changes to handle this edge case more comprehensively by mid next week. * More aggressive monitoring to detect delays in “SIP 100 Trying” responses is being worked on.