Retell AI experienced a major incident on September 5, 2026 affecting Web Call and Phone Call, lasting 1h 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 05, 2026, 11:49 PM UTC
We are investigating an issue causing web and outbound calls to fail and inbound calls to experience connection delays. We will provide an update as soon as more information is available.
- resolved Sep 06, 2026, 12:19 AM UTC
The issue affecting web, outbound, and inbound calls has been resolved. All calling services are operating normally.
- postmortem Sep 06, 2026, 03:35 AM UTC
### Summary On September 5, a rare race condition in our custom LLM call path caused call scheduling queues across our fleet to stall. Because the failure also made containers appear underutilized, autoscaling never triggered. Between roughly 10:00 and 16:00 PST the impact grew until most containers were stuck and new calls could not be placed. ### Impact * **Web calls, outbound calls, and inbound calls** were all affected. * Inbound calls were the least affected, since they hold the highest priority in our scheduling queue. * Peak impact occurred around 16:00 PST. ### Root Cause The failure required a specific sequence of events: 1. **Stalled initialization.** If a call ended before its custom LLM websocket was established, the call's initialization never completed. The scheduling queue waited indefinitely for that initialization to finish. 2. **Queue exhaustion.** Each container allows a fixed number of concurrent dequeue operations. Enough stalled initializations consumed all available slots, leaving the container unable to pick up new calls. 3. **Misleading scaling signals.** The call-end path still ran and decremented the container's concurrency counter. As a result, stuck containers reported low concurrency and low resource utilization — so no scale-out was triggered. With every container blocked and autoscaling idle, new calls had nowhere to go. This pattern is only reachable through that exact ordering, which is why it went undetected until it occurred at scale. ### Timeline \(PST\) ### Prevention ### Closing We apologize for the disruption. We are treating this as an opportunity to harden our scheduling and scaling infrastructure, and we will ship the remaining action items within a few days.