Latency observed in CSX for US Swimlanes Production Environment

Update timeline

1. resolved Feb 26, 2026, 03:51 PM UTC

   The CSOD Technology Team observed a performance degradation affecting all US swimlanes (Production Environment). The problem began at 6.10 am Pacific Time and was resolved at 6.29 am Pacific Time. During this time, clients with portals on these swimlanes may have experienced slow performance or intermittent errors while accessing the application. The issue has been identified and resolved, and services have been restored to normal operations. Thank you for your patience and understanding!

2. postmortem Mar 15, 2026, 06:58 PM UTC

   **Issue Summary:** Between February 26 and March 12, 2026, users experienced intermittent latency while accessing the platform hosted in the US SL1, SL2, SL3, and SL5 environments. During this period, some requests took longer than expected to process, resulting in degraded performance across the application. The issue impacted multiple swimlanes but did not result in complete service unavailability. **Root Cause:** The latency was caused by stability issues within an underlying authentication service responsible for user authentication and session management. Increased authentication activity, partly driven by the enablement of global SSO capabilities for multiple customers, placed additional load on the service. This resulted in intermittent service instability and elevated request processing times across the affected environments. **Corrective Action:** Cornerstone Engineering teams implemented the following actions to restore service stability: * Increased authentication service capacity to reduce immediate load and stabilize the authentication infrastructure. * Conducted a detailed review of the service implementation to identify inefficiencies. * Optimized the service by removing unnecessary code references and improving memory management. Following these actions, the authentication service stabilized and platform latency returned to normal levels. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone has taken the following actions: * Increased authentication service capacity to better support growing authentication demand. * Implemented code and memory management optimizations to improve service stability and efficiency. * Introduced enhanced monitoring to provide greater visibility into authentication service performance and detect anomalies earlier.