RETROACTIVE: [Sev 2] OTP Verifications Degradation

Update timeline

1. resolved Jun 15, 2026, 05:37 AM UTC

   On June 15, 2026, between 12:15 AM and 1:00 AM EDT, the OTP Verification endpoint experienced elevated error rates. The system recovered automatically at 1:00 AM EDT. No action is required from customers. Our investigation confirmed the issue was transient. All systems are operating normally. A Root Cause Analysis (RCA) will be published once our investigation is complete.

2. postmortem Jun 22, 2026, 09:39 PM UTC

   ## **Root Cause Analysis \(RCA\)** **Incident:** Elevated 5XX errors on the OTP Verification endpoint **Date Range:** June 15, 2026 — 12:15 AM ET to 1:00 AM ET **Impact:** Customers calling the Socure OTP Verification endpoint \(otp/send\) on the ID\+ production platform or RiskOS workflows experienced HTTP 5XX errors during the incident window, resulting in failed OTP verifications. The endpoint auto-recovered without manual intervention, and no customer action was required. **Status:** Resolved ### **1. Summary** On June 15, 2026, between approximately 12:15 AM and 1:00 AM ET, Socure’s OTP Verification endpoint \(otp/send\) on the ID\+ production platform experienced an elevated rate of HTTP 5XX errors. OTP requests failed or were degraded during this 45-minute window. The endpoint auto-recovered at 1:00 AM ET with no manual intervention required. A concentrated surge in OTP verification volume placed sustained load on Socure’s OTP service, which coordinates OTP generation, delivery, and verification. The service is configured to scale horizontally, but the additional replicas did not successfully come up during the surge window. The existing instances became saturated and restarted repeatedly under load, and resulted in HTTP 5XX errors for the OTP Verification endpoint during this period. Initial investigation considered upstream messaging-provider behavior as a potential contributor; subsequent log and metric analysis confirmed that vendor-side rate limiting was not a material factor during the window, and identified the internal OTP service scaling failure as the primary cause. The service self-recovered as upstream demand returned to baseline. Socure engineering has identified a set of corrective and preventive actions — described below — to address the underlying scaling behavior, strengthen capacity baselines for elevated traffic events, and improve observability around scaling-related saturation conditions. ### **2. Timeline** | Time \(ET\) | Event | | --- | --- | | 12:15 AM | Elevated HTTP 5XX error rates begin on the OTP Verification endpoint \(otp/send\) in the ID\+ production environment. | | 12:17 AM – 12:28 AM | Initial high-volume burst observed on the OTP Verification endpoint; OTP verification request volume rose sharply, placing concentrated load on the MFA Orchestrator service that coordinates OTP delivery and verification. | | 12:41 AM | Incident formally declared and opened within Socure’s incident management system. On-call engineering paged; incident response team assembled. An investigation bridge was convened and telemetry for the OTP service was reviewed in parallel. | | ~12:45 AM | Initial investigation considered upstream messaging-provider behavior as a potential contributor; no broad vendor outage reported on the public status page. Investigation continued in parallel into the health of Socure-owned services in the OTP path. | | 1:00 AM | Upstream OTP verification request volume subsided; service load on the OTP service returned to capacity, and error rates on the OTP Verification endpoint auto-recovered to normal levels. No manual mitigation was required. | | Post Incident | Incident moved to post-incident review; engineering continued investigation and impact assessment to confirm scope of affected accounts and validate the upstream cause. | | June 16 | Detailed log and metric analysis completed. Primary contributor identified as a horizontal scaling failure in Socure’s OTP service: the configured autoscaling did not produce additional replicas during the surge, the existing instances became saturated, and the majority of the observed errors traced to OTP service response failures. Upstream messaging-provider rate limiting was ruled out as a material factor. Investigation into the underlying autoscaling behavior is ongoing with Site Reliability Engineering. | ### **3. Root Cause** **Primary Root Cause:** A concentrated surge in OTP verification request volume placed a sustained load on Socure’s OTP service — the service responsible for coordinating one-time passcode generation, delivery, and verification. The service is configured to scale horizontally under load. During the surge window, the additional instances did not successfully come up. The existing replicas became saturated and restarted repeatedly under sustained load, and the majority of the HTTP 5XX errors observed traced to response failures during this saturation period. Subsequent log and metric analysis confirmed that upstream messaging-provider rate limiting was not a material factor during the window. The dominant error signature in OTP logs corresponded to internal response-handling failures under load, not to vendor-side throttling. The condition was transient: as upstream OTP request volume returned to baseline, pod load returned to capacity and the OTP Verification endpoint auto-recovered without any change to Socure-owned services or vendor configuration. **Contributing Factors:** * **Horizontal scaling failure:** The service’s configured horizontal autoscaling policy did not produce additional instances during the surge window. Without the headroom of additional instances, the existing instances were unable to sustain throughput, became saturated, and restarted multiple times. Investigation into the underlying cause of the autoscaling failure is ongoing with Site Reliability Engineering. * **Baseline capacity headroom:** The baseline instance count for the OTP service, combined with the time required to bring additional instances online, did not provide sufficient headroom to absorb the observed surge prior to additional capacity becoming available. **Non-contributing Factors:** * **Upstream messaging provider:** Service-level rate limiting from the upstream messaging provider was not a material factor during the incident window. Genuine vendor-side 4XX responses observed in logs were limited in volume — far below the dominant OTP error — and did not drive the customer-visible impact. * **Recent changes or deployments:** No Socure application code, configuration, or release change was associated with the incident window. No broad outage was reported on any third-party dependency’s public status page during this period. ### **4. Resolution** * **Service restoration:** The condition was transient and self-resolved as upstream OTP verification request volume returned to baseline. Load on the OTP service returned to capacity, and error rates on the OTP Verification endpoint returned to normal at approximately 1:00 AM ET on June 15, 2026. No manual mitigation was required. Socure engineering verified recovery through telemetry and continued monitoring through and beyond the incident window. * **Validation and verification:** A post-incident review was conducted to confirm the scope and duration of impact. Detailed log and metric analysis identified the OTP service horizontal scaling failure as the primary contributor and ruled out upstream messaging-provider rate limiting as a material factor. Investigation into the underlying cause of the autoscaling failure is ongoing with Socure’s Site Reliability Engineering team. ### **5. Corrective and Preventive Actions** | # | Action | Priority | Status | | --- | --- | --- | --- | | 1 | Vendor engagement — Confirm with the upstream messaging provider the applicable API rate-limit model and validate the effective limits in force for Socure’s account. | High | In Progress | | 2 | Capacity baseline review — Re-baseline the OTP capacity envelope; revisit the baseline \(pre-scale\) instances count and autoscaling thresholds to ensure sufficient headroom to absorb peak surge traffic prior to additional replicas becoming available. | High | ETA: June 25th | | 3 | OTP autoscaling investigation — Investigate with Site Reliability Engineering the root cause of why the configured horizontal autoscaling did not produce additional replicas during the surge, covering pod scheduling, resource constraints, and node availability. | High | In Progress | | 4 | Capacity planning and forecasting — Establish a recurring process to forecast peak OTP send volume from high-throughput customer programs and pre-coordinate vendor capacity ahead of anticipated traffic events. | Medium | Planned | ### **6. Conclusion** This incident was caused by a horizontal scaling failure in Socure’s OTP service during a brief, concentrated surge in OTP verification traffic. The service’s configured autoscaling policy did not produce additional instances during the surge window, the existing instances became saturated, and the majority of the errors observed traced to OTP service response failures. The condition was transient and self-resolved as upstream traffic subsided. Subsequent investigation ruled out upstream messaging-provider rate limiting as a material factor, and engineering is actively investigating the underlying cause of the autoscaling failure to prevent recurrence. Socure takes accountability for this incident and the impact it had on the affected customers. We remain committed to improving the resilience and maintaining a reliable experience for our customers.