Bland experienced a critical incident on April 13, 2026 affecting API, lasting 1h 35m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 13, 2026, 02:14 PM UTC
We are observing heightened API timeouts, affecting both inbound and outbound calling. We are actively investigating the alarms.
- monitoring Apr 13, 2026, 02:31 PM UTC
Error rate has decreased, calls and dashboard access are operational again. Continuing to monitor as we investigate root cause.
- resolved Apr 13, 2026, 03:10 PM UTC
API is stable, a root cause at the database level has been identified. Once confirmed, postmortem will be updated with findings.
- postmortem Apr 13, 2026, 05:38 PM UTC
# **API Service Disruption — April 13, 2026** **Resolved** | Impact: 06:34–07:22 PDT \(48 minutes\) | Severity: Major ## **What happened** For approximately 48 minutes on the morning of April 13, our API was degraded. Most affected requests timed out on the client side after 60 seconds or returned HTTP 5xx errors; voice-call orchestration, SMS send, and webhook callbacks were the most visibly impacted. The service recovered on its own at 07:22 PDT. No data was lost. ## **Cause** A chronic database condition had been accumulating silently for six days: a background replication process prevented our primary Postgres database from reclaiming space left by updated rows, and our busiest update-heavy table gradually developed significant bloat. On the morning of April 13, the resulting slowdown was enough to saturate our API-side connection pools. API workers blocked waiting for the database, new requests queued behind them and hit the 60-second timeout, and the backlog took 40 minutes to drain. ## **What we're doing** * Cleared the stuck replication state and scheduled database maintenance on the affected tables. * Adding monitoring to catch this chronic condition days in advance. * Adding an API-level circuit breaker so a slow database cannot cascade into a stopped API again. * Rewriting the slow query patterns that reduced our operational headroom. We're sorry for the disruption. If this incident affected your workload and you'd like a more detailed discussion, please reach out to your account contact.