LiveKit incident
Disconnections and failed connections in multiple regions
Resolved after 1 hour 39 minutes, affecting US East - Ingress and US West - Ingress and 1 more, on October 31, 2026.
This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed. - About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed. The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC. Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.
Affected components
Update timeline
- resolved Oct 31, 2026, 04:05 PM UTC
This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed. - About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed. The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC. Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.
Your product runs on more than LiveKit
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services