Unintended Credits deduction

Update timeline

1. identified Jul 16, 2025, 07:58 AM UTC

   We have identified an issue affecting a subset of customers, where credit deductions are occurring at a higher rate than intended. Our team is actively investigating and working to resolve the problem.

2. monitoring Jul 16, 2025, 07:59 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jul 16, 2025, 07:59 AM UTC

   This incident has been resolved.

4. postmortem Jul 16, 2025, 08:23 AM UTC

   **Date:** 2025-07-11 **Duration:** 12:00 - 12:30 UTC ### **Affected Services** * Outbound/Inbound Calling * SMS Delivery * IQ Transcriptions * Other usage-based services relying on balance deduction logic ### **Impact** A subset of customers experienced **excessive and inaccurate balance deductions** across multiple services including: * **Outbound calls** * **SMS messaging** * **IQ and transcription usage** This incorrect deduction affected customer billing visibility and usage quotas, resulting in: * Unexpected balance exhaustion * Interrupted service usage due to perceived insufficient balance ### **Root Cause** The incident was caused by **flawed cache key reuse** in the currency conversion logic used during balance deduction operations: * Cache entries related to currency conversion rates \(e.g., USD to other currencies\) were **improperly reused across services**. * As a result, **USD-denominated prices were incorrectly multiplied by cached exchange rates** meant for other contexts or currencies. * This led to **multiplied deductions**, far beyond actual usage cost. ### **Resolution** * Engineers immediately turned on the circuit breaker to ensure customers are able to use Calling, SMS and other services. * Engineers identified the faulty cache key reuse in the balance deduction service. * Flushed the cache to restore cache integrity * Affected customer balances were audited and **refunds/credits processed** where applicable.