RiskOS Service Degradation

Update timeline

1. investigating Jun 23, 2026, 02:56 AM UTC

   We are currently experiencing errors affecting our RiskOS APIs. Our team is actively investigating the issue and will provide updates as more information becomes available.

2. monitoring Jun 23, 2026, 03:07 AM UTC

   Our team has mitigated the issue that was causing errors with the RiskOS APIs. We are seeing error rates return to normal levels and are continuing to monitor the system closely. We will provide additional updates if there are any further changes.

3. monitoring Jun 23, 2026, 03:24 AM UTC

   We are continuing to monitor our services and are not observing any errors at this time. We will provide additional updates if there are any further changes.

4. resolved Jun 23, 2026, 03:29 AM UTC

   The issue impacting RiskOS has been resolved, and service has returned to normal operation. Customers should no longer experience errors, degraded performance, or delays when accessing RiskOS services and workflows. Our teams have confirmed system stability and will continue to monitor performance closely. We appreciate your patience throughout this incident and apologize for the impact this outage may have had on your operations.

5. postmortem Jun 26, 2026, 09:07 PM UTC

   **Root Cause Analysis** **Incident:** RiskOS Evaluation API Elevated 502 Errors **Date Range:** June 22, 2026, 10:23 PM EDT – 10:59 PM EDT **Impact:** Customers calling the Socure RiskOS Evaluation APIs in production experienced an elevated rate of HTTP 502 \(Bad Gateway\) responses for approximately 36 minutes. **Status:** Resolved ## **1. Summary** On June 22, 2026, between approximately 10:23 PM and 10:59 PM EDT, Socure’s RiskOS Evaluation APIs in the production environment experienced an elevated rate of HTTP 502 errors. The disruption was caused by a misconfiguration in the aggregations engine — a component that enriches Evaluation API processing by computing velocity based signals across entity data. The misconfiguration was introduced during a configuration update as part of Local Graph GA readiness, and was not caught by existing validation controls. When the service attempted to use this configuration during evaluation processing an unexpected runtime exception occurred that triggered automated service restarts. As the affected services cycled, backend capacity became intermittently unavailable, causing requests during those periods to receive HTTP 502 \(Bad Gateway\) responses. Socure's automated monitoring detected the elevated error rate at **10:24 PM EDT** and the Engineering team began investigation immediately. The team mitigated the issue by disabling aggregations processing to stabilize traffic, remediated the invalid configuration, and validated recovery before re-enabling aggregations. Evaluation APIs returned to normal operation by **10:59 PM EDT**. Aggregations were re-enabled after validation at approximately **11:13 PM EDT**. Requests that received HTTP 502 responses during the incident window were not processed and no corresponding evaluation was created. Customers who experienced HTTP 502 responses between 10:23 PM and 10:59 PM EDT on June 22, 2026 should verify any intended evaluations and resubmit them if necessary. ## **2. Timeline** | **June 22, 2026 / Time** | **Event** | | --- | --- | | **10:22 PM EDT** | A configuration change in Entity Aggregations introduced an invalid value. | | **10:23 PM EDT** | Elevated HTTP 502 error rates begin on RiskOS Evaluation APIs in the production environment. | | **10:24 PM EDT** | Automated monitoring detected the elevated error rate and the on-call Engineering team was notified. Investigation began immediately. | | **10:25 PM to 10:40 PM EDT** | Engineering observed multiple servers failing intermittently. The team reviewed logs and traced the failures across affected servers to isolate the root cause. | | **~10:40 PM EDT** | Engineering identified that the issue was related to an aggregations configuration affecting Evaluation API processing. | | **~10:45 PM EDT** | Engineering began targeted mitigation, initially isolating specific aggregations-dependent components. Instability persisted, and the team proceeded to disable aggregations processing entirely to stabilize traffic. | | **10:55 PM EDT** | Aggregations processing was fully disabled. Evaluation API traffic stabilized. | | **10:59 PM EDT** | Evaluation APIs returned to normal operation. Elevated 502 errors were resolved. | | **10:59 PM to 11:13 PM EDT** | Engineering identified and remediated the specific invalid aggregation configuration. Recovery was validated against the corrected configuration. | | **11:13 PM EDT** | Aggregations re-enabled following successful validation of the corrected configuration. | ## **3. Root Cause** #### **Primary Root Cause:** An invalid configuration was applied to the aggregations engine. When the service attempted to use this configuration during Evaluation processing, it triggered an unhandled failure path inside the processing pipeline. Rather than gracefully rejecting the invalid configuration, the failure path caused service-level crashes and the affected servers entered a restart cycle. While the affected services were cycling, the RiskOS Evaluation APIs' available capacity to serve traffic dropped, and a portion of incoming requests reached the gateway without a healthy backend service available to respond. The gateway returned HTTP 502 \(Bad Gateway\) responses for those requests. #### **Contributing Factors:** * **Insufficient configuration validation:** The server did not gracefully reject the invalid Aggregation configuration at the load time. Instead of failing safely, the invalid spec propagated into a runtime failure path that crashed the affected pod, producing the customer-visible 502 responses. * **Shared processing impact:** Because the issue affected shared Evaluation API processing, a configuration-specific issue resulted in elevated API failures for customers during the incident window. #### **Non-contributing Factors:** * **Infrastructure health:** No underlying cluster, network, storage, or platform infrastructure issue contributed to the incident. Compute, network, and storage layers supporting RiskOS remained healthy throughout the incident window. Once the offending configuration was mitigated, the backend service stabilized with Evaluation processing restored back to normal. ## **4. Resolution** Socure restored service by disabling aggregations processing entirely, which stabilized Evaluation API traffic and returned normal operation by 10:59 PM EDT. The team then identified and removed the specific invalid aggregation configuration, validated the correction, and re-enabled aggregations at 11:13 PM EDT. Runtime safeguards have been implemented to handle invalid aggregation configurations gracefully without affecting service availability. Pre-activation validation checks currently in development will prevent invalid configurations from reaching production. ## **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | **Runtime safeguards for Aggregations Configurations** | Updated Evaluation API processing to handle incomplete or missing aggregation configuration values gracefully across all similar configuration processing paths, preventing service crashes. The specific condition that triggered this incident is now handled safely without impacting service stability. | **Completed** | | **Aggregations configuration validation** | Added stricter validation so invalid, incomplete, or unsupported configurations - are rejected before they can become active in production. | **In Progress** | | **Pre-activation configuration checks** | Added checks before aggregation configurations are used in Evaluation API processing to identify invalid or conflicting states earlier in the pipeline, before they can impact live traffic. | **In Progress** | | **Blast radius control** | Strengthen isolation so an issue in a specific workflow path does not affect broader shared Evaluation API processing. | **Q3, 2026** | ## **6. Lessons Learned** This incident highlighted a specific gap between Aggregations configuration validation and Evaluation API runtime handling. Follow-up work is focused on preventing invalid configurations from becoming active, improving runtime safeguards, and strengthening isolation for workflow-specific issues. ## **7. Next Steps & Ongoing Commitment** Socure recognizes that customers rely on RiskOS Evaluation APIs for reliable, real-time decisioning. We take ownership of the disruption caused by this incident and are implementing changes to reduce the likelihood and impact of similar issues. The corrective actions underway focus on three areas: strengthening aggregations configuration validation, improving Evaluation API runtime safeguards, and limiting the blast radius of any future configuration-related issues. We remain committed to improving the resilience and maintaining a reliable experience for our customers.