RETROACTIVE: [SEV 1] Outage Resulting in 5xx Errors

Update timeline

1. resolved Oct 16, 2026, 02:23 AM UTC

   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:12 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.

2. postmortem Oct 17, 2026, 01:42 AM UTC

   ## **Root Cause Analysis** **Incident:** Socure APIs Outage **Date Range:** 15-Oct-2026 **Impact:** Interruption in service for Socure APIs **Status:** Resolved ### **1. Summary** On 15-Oct-2026, between 18:12 and 19:00 Eastern time Socure experienced an interruption to its RiskOS and ID\+ APIs, causing errors and timeouts to users. The first phase ran from 18:12 to 18:20, and was a major interruption for virtually every service. The second phase of the outage ran from 18:20 to 19:00, and was a partial outage for services that rely on KYC data, where customers saw elevated error rates and latencies for some services, but traffic was still being served. ### **2. Timeline** ### **3. Root Cause** * **Primary Root Cause:** As a part of an ongoing migration, an on-call SRE made a change to ArgoCD, the system we use to manage the kubernetes clusters Socure runs its services on. The change was scoped to a single cluster, was reviewed carefully before pushing, caused no issues in other environments it was tested in, and should have had no visible impact. Unfortunately, the change updated a legacy setting that did not appear in the change review. That legacy setting governed what services ArgoCD managed, and removed many live services from ArgoCD. In consequence, ArgoCD removed those services from production briefly, before the setting was corrected. * **Contributing Factors:** The KYC Search service is a backend piece of Socure infrastructure that provides services to several of our internal products. When that service restarted from scratch at the same time as all of the clients of the service did, a small number of pods in the service crashed under load. This increased the load on the remaining pods, some of which crashed in turn. This cascading failure prevented the service from coming back on its own the way the rest of Socure’s infrastructure did. Diagnosing this separate failure mode took additional time, and remediating it fixed the remaining errors after the initial incident. ### **4. Resolution** The ArgoCD change was reverted immediately, causing services to come back to their prior state. The KYC Search service required manual effort to block traffic from reaching it until a majority of pods in the service could come up. After the service was healthy, we could restore traffic to it and resolve the overall incident. ### **5. Corrective and Preventive Actions** ### **6. Lessons Learned** * Socure’s tooling for viewing the changes it makes to production did not work correctly. We are reviewing these tools to ensure they operate correctly under all use cases and fix any cases like this where the full scope of a change is not clear. * Augment the existing change review process with agentic support. While in this case, the specific change was not visible in the plan, having an extra reviewer that is supported by an extensive training model will help ensure issues are caught before submission. * Even changes that appear to have no impact, when they are made to the governance system for our production jobs, can be risky, and should be shifted further out of US business hours. * Our KYC Search service is vulnerable to a “thundering-herd” problem when it receives full traffic flow when it is not fully online. We need to strengthen our circuit breakers to prevent this issue recurring. * We run too many of our core API services in a single kubernetes cluster. Even though that cluster is distributed across multiple availability zones, the shared control plane creates too high a blast radius for changes to that cluster. ### **7. Next Steps & Ongoing Commitment** Socure understands our customers rely on us as a key provider in critical business flows, and that this service interruption is not acceptable. We apologize for the impact this issue caused, and commit to ensuring it does not recur. To that end, we will deliver details on our plans to re-architect our application layout by November, as well as improve the resilience of the KYC Search service in Q1 of next year.