Retell AI Outage History
Retell AI is up right nowPingoru has recorded 51 incidents for Retell AI: 28 in the last 12 months and 4 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 1 of 720 (0%) |
| May 2026 | 4 | 4 of 744 (0%) |
| June 2026 | 3 | 13 of 720 (2%) |
| July 2026 | 3 | 4 of 744 (0%) |
| August 2026 | 1 | 0 of 744 (0%) |
| September 2026 | 2 | 2 of 720 (0%) |
| October 2026 | 0 | 0 of 512 (0%) |
Incidents, newest first
Major September 17, 2026 Call service disruption 19m Web Call, Phone Call
Started Sep 17, 2026, 01:15 AM UTC· resolved Sep 17, 2026, 01:35 AM UTC
- investigating Sep 17, 2026, 01:15 AM UTC
We are aware of issues with some calls and are investigating.
- resolved Sep 17, 2026, 01:35 AM UTC
We have found the issue and resolved it. About 10% of calls may have been affected and have no audio.
- postmortem Sep 17, 2026, 02:30 AM UTC
**Status: resolved.** All times are PST. ## Summary On September 16 at 5:37 PM we deployed a planned improvement to call handling. The improvement was not meant to be active yet, but the deployment turned it on, and it began to assign live calls to workers. This change unintentionally let two workers take the same call, and each worker then started it. The two starts interfered with each other, and the call failed. About 22% of calls were affected from 5:37 PM to 6:20 PM. We stopped the new workers, rolled the deployment back, and saw the issue resolve. ## Impact An affected call shows one of two signs: * The call ends as soon as it connects. The duration is zero or close to zero seconds. * The call connects. The agent does not speak. The call ends after a silence. A call that ran correctly is not affected. Its recording, its transcript, and its billing record are correct. ## Root cause We were making an improvement to our system for call handling. As part of this change, we made a new queue for calls and a new set of workers, which was meant to be inactive. Our old servers were writing to both queues as a test, which did not cause issues as the new workers were not active. A deployment replaces servers one group at a time, so when the deployment began at 5:37PM, some servers ran the old code and some ran the new code. The old servers still wrote each call to both queues, and the new workers now read the new queue. The two systems therefore handled the same call, once each, and undid each other's work. ## Timeline \(PST\) | Time | Event | | --- | --- | | 5:37 PM | We deploy the change. | | 5:40 PM | Call errors increase. | | 5:43 PM | Our call monitor reports the first failures. | | 5:47 PM | The system pages the on-call engineer. | | 6:14 PM | We stop the new call workers. Double handling drops sharply. | | 6:18 PM | We start the rollback, which removes the change itself. | | 6:20 PM | The last affected call. Call errors return to normal. | | 7:54 PM | The rollback finishes. | ## Prevention 1. \(IN PROGRESS\) **Improve rollout process checks** We will improve our process checks for large changes to ensure that future deployments roll out in the intended state, and explicitly call out deployment time checks. 2. \(IN PROGRESS\) **Let the monitor stop a deployment.** We will connect our call monitor to our deployment system. Repeated failures will then stop a deployment. ## Closing We are sorry for the disruption. A failed call is a bad experience for you, and for the person you called.
Major September 5, 2026 Web Calls and Phone Calls Disruption 1h 8m Web Call, Phone Call
Started Sep 05, 2026, 11:10 PM UTC· resolved Sep 06, 2026, 12:19 AM UTC
- investigating Sep 05, 2026, 11:49 PM UTC
We are investigating an issue causing web and outbound calls to fail and inbound calls to experience connection delays. We will provide an update as soon as more information is available.
- resolved Sep 06, 2026, 12:19 AM UTC
The issue affecting web, outbound, and inbound calls has been resolved. All calling services are operating normally.
- postmortem Sep 06, 2026, 03:35 AM UTC
### Summary On September 5, a rare race condition in our custom LLM call path caused call scheduling queues across our fleet to stall. Because the failure also made containers appear underutilized, autoscaling never triggered. Between roughly 10:00 and 16:00 PST the impact grew until most containers were stuck and new calls could not be placed. ### Impact * **Web calls, outbound calls, and inbound calls** were all affected. * Inbound calls were the least affected, since they hold the highest priority in our scheduling queue. * Peak impact occurred around 16:00 PST. ### Root Cause The failure required a specific sequence of events: 1. **Stalled initialization.** If a call ended before its custom LLM websocket was established, the call's initialization never completed. The scheduling queue waited indefinitely for that initialization to finish. 2. **Queue exhaustion.** Each container allows a fixed number of concurrent dequeue operations. Enough stalled initializations consumed all available slots, leaving the container unable to pick up new calls. 3. **Misleading scaling signals.** The call-end path still ran and decremented the container's concurrency counter. As a result, stuck containers reported low concurrency and low resource utilization — so no scale-out was triggered. With every container blocked and autoscaling idle, new calls had nowhere to go. This pattern is only reachable through that exact ordering, which is why it went undetected until it occurred at scale. ### Timeline \(PST\) ### Prevention ### Closing We apologize for the disruption. We are treating this as an opportunity to harden our scheduling and scaling infrastructure, and we will ship the remaining action items within a few days.
Minor August 7, 2026 Some inbound calls failing to connect 26m Phone Call
Started Aug 07, 2026, 09:55 PM UTC· resolved Aug 07, 2026, 10:22 PM UTC
- identified Aug 07, 2026, 09:55 PM UTC
Due to issues with an upstream provider, we are seeing connection issues for a portion of inbound calls. This is not widespread.
- monitoring Aug 07, 2026, 10:16 PM UTC
Service should be back to normal. We are continuing to monitor to ensure stability.
- resolved Aug 07, 2026, 10:22 PM UTC
This incident has been resolved.
Major July 29, 2026 Inbound calling not connecting 1h 41m Phone Call
Started Jul 29, 2026, 02:00 PM UTC· resolved Jul 29, 2026, 03:42 PM UTC
- investigating Jul 29, 2026, 03:26 PM UTC
We identified that certain number's inbound is not connecting, investigating. Outbound is not impacted.
- identified Jul 29, 2026, 03:35 PM UTC
The issue was identified as a datacenter issue from underlying telephony provider, where 2 out of 3 datacenters failed to deliver webhook to Retell backend server.
- monitoring Jul 29, 2026, 03:39 PM UTC
This has been fixed, we are monitoring.
- resolved Jul 29, 2026, 03:42 PM UTC
This incident has been resolved.
Notice July 14, 2026 Batch calls failing 49m
Started Jul 14, 2026, 03:42 PM UTC· resolved Jul 14, 2026, 04:32 PM UTC
- investigating Jul 14, 2026, 03:42 PM UTC
Outbound batch calls are failing. We are investigating the issue
- investigating Jul 14, 2026, 04:32 PM UTC
We are continuing to investigate this issue.
- resolved Jul 14, 2026, 04:32 PM UTC
This incident has been resolved.
Notice July 1, 2026 TTS provider 11labs v3 model is down 1h 8m
Started Jul 01, 2026, 09:45 PM UTC· resolved Jul 01, 2026, 10:53 PM UTC
- identified Jul 01, 2026, 09:45 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Jul 01, 2026, 10:53 PM UTC
This incident has been resolved.
Major June 24, 2026 partial outage affecting incoming and ongoing call connections 1h 33m Phone Call
Started Jun 24, 2026, 04:31 PM UTC· resolved Jun 24, 2026, 06:05 PM UTC
- monitoring Jun 24, 2026, 04:31 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 24, 2026, 06:05 PM UTC
This incident has been resolved.
Minor June 22, 2026 Chats and calls unavailable for GPT-4o and 4o-mini 2h 3m Web Call, SMS chat, Phone Call, API chat
Started Jun 22, 2026, 07:00 AM UTC· resolved Jun 22, 2026, 09:03 AM UTC
- identified Jun 22, 2026, 08:01 AM UTC
Due to a recent regression, AI calls and chats powered by GPT-4o and GPT-4o mini are currently unavailable. Our team is working to restore service as quickly as possible.
- monitoring Jun 22, 2026, 08:16 AM UTC
A fix is being deployed; we are actively monitoring.
- resolved Jun 22, 2026, 09:03 AM UTC
The incident has been resolved
Minor June 12, 2026 Platform voice Grace and Nico not working 9h 13m Web Call, Phone Call
Started Jun 12, 2026, 07:30 AM UTC· resolved Jun 12, 2026, 04:44 PM UTC
- investigating Jun 12, 2026, 04:31 PM UTC
The platform voice, Grace and Nico, are not working
- resolved Jun 12, 2026, 04:44 PM UTC
The issue is resolved now
Minor May 28, 2026 Certain calls are not being picked up in time 1h 29m Web Call, Phone Call
Started May 28, 2026, 03:00 PM UTC· resolved May 28, 2026, 04:30 PM UTC
- investigating May 28, 2026, 04:37 PM UTC
Many calls are experiencing high latency, no pickup, or being dropped due to web socket connection issues. We are investigating now.
- resolved May 28, 2026, 05:52 PM UTC
This incident has been resolved. It was due to a connection issue in one of the data centers for the telephony provider. We are adding more fallback routes there.
Major May 25, 2026 Calls returning concurrency_limit_reached 40m Web Call, Phone Call
Started May 25, 2026, 11:00 PM UTC· resolved May 25, 2026, 11:41 PM UTC
- investigating May 25, 2026, 11:00 PM UTC
Due to an error in the code, the concurrency of orgs are incorrectly calculated and returning concurrency_limit_reached for many calls.
Major May 25, 2026 Call/chat history, analytics and QA dashboard down 30m Alerting, Analytics, Dashboard, QA
Started May 25, 2026, 09:09 PM UTC· resolved May 25, 2026, 09:40 PM UTC
- investigating May 25, 2026, 09:09 PM UTC
Due to a database outage, call/chat history, analytics and QA dashboard down are currently completely down. We are working with the database team to fix the issue.
- investigating May 25, 2026, 09:12 PM UTC
We are continuing to investigate this issue.
- investigating May 25, 2026, 09:13 PM UTC
We are continuing to investigate this issue.
- identified May 25, 2026, 09:33 PM UTC
We are rolling out a fix to direct the traffic to our database replica while the provider is investigating the issue. Note that no call or chat data were lost.
- resolved May 25, 2026, 09:40 PM UTC
The issue has been fixed and all services are back to working.
Major May 8, 2026 Agent reading and publishing broken 1h
Started May 08, 2026, 03:00 AM UTC· resolved May 08, 2026, 04:00 AM UTC
- resolved May 08, 2026, 04:50 PM UTC
agent publishing was broken for ~1 hour - returned server error. Calling was not affected
Minor April 13, 2026 KB Retrieval and QA Failure 28m QA, Knowledge Base
Started Apr 13, 2026, 11:47 PM UTC· resolved Apr 14, 2026, 12:15 AM UTC
- monitoring Apr 13, 2026, 11:47 PM UTC
The KB retrieval and QA experienced an issue due to an unintended code change. The issue has been resolved, and we are actively monitoring the systems to ensure continued stability.
- resolved Apr 14, 2026, 12:15 AM UTC
This incident has been resolved.
Minor March 16, 2026 Batch calls not sent 40m Phone Call
Started Mar 16, 2026, 06:41 PM UTC· resolved Mar 16, 2026, 07:21 PM UTC
- investigating Mar 16, 2026, 06:41 PM UTC
Starting ~13:32 UTC (8:32 AM PT), batch calls were not being sent due to an internal issue that was triggered in our system. We've identified the root cause and are actively working on a fix.
- monitoring Mar 16, 2026, 07:14 PM UTC
A fix is being deployed and we are monitoring to ensure recovery. As a side effect, batch calls that were missed are being marked as "sent".
- resolved Mar 16, 2026, 07:21 PM UTC
This incident has been resolved.
Minor March 16, 2026 Custom LLM calls failing to start 14m Web Call, Phone Call
Started Mar 16, 2026, 04:39 AM UTC· resolved Mar 16, 2026, 04:54 AM UTC
- identified Mar 16, 2026, 04:39 AM UTC
19:38 PST: custom LLM calls started failing to start due to a bad release. Other types of calls are not impacted. 21:35 PST: a fix is being rolled out, we are keep monitoring the issue.
- resolved Mar 16, 2026, 04:54 AM UTC
This incident has been resolved.
Notice March 4, 2026 MiniMax TTS issue – Web Call, Phone Call
Started Mar 04, 2026, 10:34 PM UTC· resolved Mar 04, 2026, 10:34 PM UTC
- resolved Mar 04, 2026, 10:34 PM UTC
Between 1:25–2:08pm PT, a provider-side rate-limit configuration issue caused some TTS requests to be silently dropped. This issue has been resolved, and we are adding detection to make sure a fallback will be triggered if this happens again.
Notice March 4, 2026 Small portion of community voice not working –
Started Mar 04, 2026, 02:30 PM UTC· resolved Mar 04, 2026, 02:30 PM UTC
- resolved Mar 04, 2026, 06:21 PM UTC
There was an accidental change in the logic of account cleanup that led to the deletion of small portion of community voice resources, which caused those agents with the voice having issues. We have fixed the logic, and have been running a backfill to detect and add back those resources.
Minor February 20, 2026 Telynx - SIP Trunking Service Degradation 2h 16m Phone Call
Started Feb 20, 2026, 09:43 PM UTC· resolved Feb 21, 2026, 12:00 AM UTC
- identified Feb 20, 2026, 09:43 PM UTC
We are currently experiencing service degradation affecting SIP trunking through our upstream provider, Telnyx. This may result in call connection issues. https://status.telnyx.com/
- resolved Feb 21, 2026, 02:01 AM UTC
This incident has been resolved.
Major February 10, 2026 Calls failing to start and concurrency stuck 5m Phone Call
Started Feb 10, 2026, 10:12 PM UTC· resolved Feb 10, 2026, 10:18 PM UTC
- identified Feb 10, 2026, 10:12 PM UTC
There were call initialization issues between 1:35pm and 1:43pm PST. Calls are back online now, but concurrency for some users may be stuck. We are working on a fix.
- resolved Feb 10, 2026, 10:18 PM UTC
Calls and concurrency should be back to normal.
Minor January 27, 2026 Scheduled batch calls not running 2h 45m
Started Jan 27, 2026, 10:30 PM UTC· resolved Jan 28, 2026, 01:15 AM UTC
- resolved Jan 28, 2026, 02:52 AM UTC
Scheduled batch calls did not run from 2:35 PM to 5:15 PM PST. The service has now recovered. Any batch calls whose scheduling window still included 5:15 PM were executed at that time. Batch calls whose scheduling window was fully missed were not executed.
Major January 22, 2026 Multilingual transcription issues 11m Web Call, Phone Call
Started Jan 22, 2026, 11:37 PM UTC· resolved Jan 22, 2026, 11:48 PM UTC
- investigating Jan 22, 2026, 11:37 PM UTC
We're aware of issues affecting multilingual transcription and are actively investigating.
- resolved Jan 22, 2026, 11:48 PM UTC
The issue appears to be resolved and multilingual transcription is recovering. We'll keep monitoring to ensure stability.
Major January 9, 2026 Outbound Calling Service Degradation 1h 56m Phone Call
Started Jan 09, 2026, 08:30 AM UTC· resolved Jan 09, 2026, 10:26 AM UTC
- investigating Jan 09, 2026, 09:26 AM UTC
Outbound calls are currently experiencing issues. Some users report receiving multiple calls from a single dial attempt, or agents being unable to hear audio. We are investigating the root cause.
- identified Jan 09, 2026, 09:58 AM UTC
The root cause has been traced to our telephony stack provider. The issue has been escalated to their support team, who are actively triaging the incident.
- monitoring Jan 09, 2026, 10:17 AM UTC
The issue should have been resolved, we are closely monitoring the status.
- resolved Jan 09, 2026, 10:26 AM UTC
The issue has been fixed
Major December 30, 2025 ASR configuration bug 44m
Started Dec 30, 2025, 09:20 PM UTC· resolved Dec 30, 2025, 10:04 PM UTC
- investigating Dec 30, 2025, 09:20 PM UTC
A recent code change introduced an issue in the ASR module that may cause some agents to go offline. We are working on a fix actively.
- investigating Dec 30, 2025, 09:20 PM UTC
We are continuing to investigate this issue.
- monitoring Dec 30, 2025, 10:00 PM UTC
The fix is being pushed to the production, we are observing and monitoring the current system status.
- resolved Dec 30, 2025, 10:04 PM UTC
This incident has been resolved.
Major December 16, 2025 Call and API failures due to Stripe outage 30m API, Web Call, Phone Call
Started Dec 16, 2025, 03:10 PM UTC· resolved Dec 16, 2025, 03:40 PM UTC
- monitoring Dec 16, 2025, 04:59 PM UTC
For around 30 minutes, calls and API requests failed due to an issue with Stripe. The outage occurred between 7:10 and 7:40 PST. This has been identified as an outage from our payment provider (Stripe). We are continuing to monitor the situation.
- resolved Dec 16, 2025, 07:56 PM UTC
This incident has been resolved.