Bland Outage History

At 11:15 AM PDT a provider side routing update caused variable extraction requests for fail for some customers. Some post call features were also affected including summary generations and analysis schemas. Fully recovered at 12:11 PM. Timeline: 11:15 AM - Requests begin erroring as the change starts rollout in provider's cluster. 11:20 AM - Rollout completes, error rates for affected models reaches 100%. 11:35 AM - Automated alarms page on call engineering, issue is raised to provider immediately. 11:45 AM - Bland engineering identifies issue as a routing change that expects a new model base URL. 11:50 AM - An override to the new base URL is applied, update begins rolling out to Bland clusters. 12:11 PM - All rollouts completed, error rates return to normal levels.

This incident has been resolved.

A misconfigured cluster of LLM replicas failed to scale up during our morning burst of traffic, causing high agent response time and possibly complete silence in severe cases between 6:00am and 6:35am PT. After diagnosing the issue we corrected our scale up mechanism to target the correct set of resources, resolving the issue.

From 2:00 PM PDT to 2:30 PM a provider outage caused limited inference capacity. Automated monitoring systems restricted outbound call dispatch during the incident. Outbound calls during this window experienced queue delays and elevated latency when they did dispatch, inbound calls were unaffected by queue delays but also experienced higher latency.

LLM response times have recovered. A postmortem will be added within the hour.

This incident has been resolved.

This incident has been resolved.

Our BTTS service experienced degraded performance due to a failure in our underlying cloud infrastructure's ability to scale compute capacity to meet demand. This resulted in agents stuttering, failing to vocalize responses, increased latency, or producing incomplete audio output during the affected window, for calls that utilize BTTS v2 voices. The timeline was as follows: 6:00 AM PST: BTTS model began failing to scale due to insufficient compute capacity in its assigned cloud cluster. 6:30 AM PST: An automated alert fired but was misconfigured as non-critical and was not actioned. ~8:00 AM PST: Rising call volume exceeded available capacity, causing noticeable impact to production calls. 9:11 AM PST: We identified degraded performance through internal monitoring and began investigating. 9:32 AM PST: BTTS compute was migrated to a separate cloud cluster with available capacity. Scaling resumed normally and performance returned to baseline.

The database provider has identified a potential internal networking configuration that may have caused the incident. Database connections and voice agent calls in Canada should be back to normal now.

This incident has been resolved.

An increase in database requests resulted in much higher latency, causing the majority of call requests (inbound and outbound) to fail from 12:56-12:59. In progress calls were unaffected.

A change to inbound call initialization caused calls to fallback to a default voice rather than the custom voice they were configured with. This impacted the majority of inbound calls from 1:52-3:18 PST.

Dialogue generation and most post call processing had high error rates from 3:09 PM - 3:49 PM PST due to a provider outage. During this period the agent was not responding/very slow to respond for some dialogue turns.

This incident has been resolved.

We will be updating this with a full resolution tomorrow morning. Currently, it looks to be due to a spike in requested data, affecting our database throughput.

An issue with an upstream inference provider caused errors in agent dialogue generation for some customers. Traffic was routed to a backup deployment as of 5:11 and error rates have recovered.

An upstream provider outage, that provides dynamic feature flag components - suffered a failure in their network configurations. We are assessing this dependency and will ensure it can no longer be a failure point for the dispatching of calls. For clarity, we attach their message below: --- 10/22/2025 17:19:00 GMT-0700 (Pacific Daylight Time) : A network issue that is impacting sdk serving configs was found. Engineering Oncall teams are currently investigating. [Update]: Incident is resolved at 5:34 PM PST.

This incident has been resolved.

This incident has been resolved.

There was an intermittent transcription issue when using Babel due to scaling with our GPU provider, causing audio to not be picked up from the user, that started at 9:30am PST. This has been fully resolved as of 11:30am PST.