Bland incident

LLM outage - Widespread agent silence

Critical Resolved View vendor source →

Bland experienced a critical incident on October 14, 2026, lasting 55m. The incident has been resolved; the full update timeline is below.

Started
Oct 14, 2026, 10:16 PM UTC
Resolved
Oct 14, 2026, 11:11 PM UTC
Duration
55m
Detected by Pingoru
Oct 14, 2026, 10:16 PM UTC

Update timeline

  1. identified Oct 14, 2026, 10:16 PM UTC

    The issue has been identified and a fix is being implemented.

  2. identified Oct 14, 2026, 10:24 PM UTC

    An upstream outage with our inference infrastructure provider has taken down our primary dialogue LLM cluster, resulting in agent silence. We have blocked further outbound dispatch and are queueing calls. We are moving workloads over to restore functionality.

  3. identified Oct 14, 2026, 10:45 PM UTC

    LLM workload is scaling up in a healthy cluster, expecting 10 minutes until resolution.

  4. monitoring Oct 14, 2026, 11:01 PM UTC

    Replicas are scaled up in the healthy cluster, monitoring system health and errors now

  5. resolved Oct 14, 2026, 11:11 PM UTC

    Replicas are healthy, opening queue back up. Expect a backlog of requests over the next hour

  6. postmortem Oct 15, 2026, 12:49 AM UTC

    **Summary** On October 14, 2026, from about 3:00 PM to 4:10 PM PT, agents on many calls went silent or took several seconds to respond. Outbound calls requested during the incident were held in a queue and placed late. The queue was fully drained by 4:41 PM PT. **What happened** A data center network outage at our upstream infrastructure provider cut off the cluster that runs the language model our agents use in conversation. The model’s time to first response rose from about 0.3 seconds to more than 7 seconds on average, so agents could not answer callers in time. To keep new outbound calls from connecting into silence, we paused outbound dispatch and queued new requests. The provider moved our model to a healthy cluster, and agents recovered by 4:10 PM PT. We reopened the queue at 4:07 PM PT and released held calls at a controlled rate. **Impact** * Live calls, inbound and outbound, from about 3:00 PM to 4:10 PM PT: agents were silent or slow to respond, and many callers hung up early. * Outbound calls requested during the incident were delayed, typically by 25 to 35 minutes. The backlog cleared by 4:41 PM PT. Calls that were canceled while queued were not placed. * We have found no impact on call records or recordings. **What we’re doing about it** * We’ve deployed a fallback that retries failed conversation requests on a second cloud provider. We’re extending it to switch providers when responses slow down, not only when they fail. * We’re spreading live-call conversation across more than one provider or cluster, so one failure can’t affect every call. * We’re changing agents to recover the turn, for example by asking the caller to repeat, instead of going silent when the model fails. * We’re paging on-call as soon as model response times rise, and slowing outbound dispatch automatically when the model is degraded. We’re sorry for the disruption. If this incident affected your calls, please contact your account team for a closer review.