Cornerstone Outage History

**Incident Summary:** Starting April 13th, 2026, intermittent errors were observed in reporting functionality across US Production swimlanes \(SL1, SL2, SL3, and SL5\). **Root Cause Analysis \(RCA\):** The issue was caused by the auto-scaling mechanism. Newly provisioned nodes were not fully ready to handle reporting requests, leading to processing delays and request timeouts. **Corrective Action:** Auto-scaling \(scale-out\) has been temporarily paused. The system is currently operating on a fixed, increased server capacity without scale-in or scale-out to ensure stability. **Preventive Actions:** * Capacity has been proactively increased to handle expected load without relying on auto-scaling. * Additional nodes are being utilized with auto-scaling disabled to maintain consistent performance. * Engineering teams are working on enhancements to improve node readiness and ensure seamless scaling in the future.

**Issue Summary:** On March 22, 2026, Edge import–related errors were observed across the US Production swimlanes \(US SL1, US SL2, US SL3, US SL5\). **Root Cause:** The issue was caused by instability within the underlying cluster infrastructure, which disrupted queue services and prevented message processing, resulting in Edge import failures across the affected swimlanes. **Corrective Action:** On March 23, 2026, emergency maintenance was performed to stabilize and reset the impacted cluster components. Once stabilized, queue processing resumed and pending Edge imports were successfully completed. **Preventive Measures:** To minimize the risk of recurrence, the cluster infrastructure was thoroughly analyzed and validated as stable following remediation.

**Incident Summary:** Since March 11th, 2026, clients have been experiencing intermittent timeouts while accessing portals across US Swimlanes \(US SL1/2/3/5\). **Root Cause Analysis \(RCA\):** The issue was attributed to congestion in the network layer, resulting in packet drops and intermittent request failures. Increased traffic load exposed capacity and routing inefficiencies across network and application tiers. ‌ **Progressive Updates & Actions Taken:** **March 11th, 2026:** * Initial issue observed with intermittent timeouts across multiple swimlanes. * Preliminary analysis indicated packet drops due to network congestion. **March 19th, 2026:** * Tune health checks and optimize scaling behavior to improve responsiveness during sudden traffic spikes. * Introduce scheduled scaling to align capacity with peak and off-hours demand. **March 26th, 2026:** * Increased number of gateway instances to distribute traffic load more effectively. * Initial improvement observed in request handling capacity. **April 1st, 2026:** * Network device configurations were refreshed to stabilize traffic flow. * Reduced packet drops observed post configuration updates. **April 7th, 2026:** * Routing updates were implemented for specific high-volume API calls to optimize traffic paths. * Latency improvements observed for targeted request flows. **April 11th, 2026:** * Network layer capacity was enhanced by adding additional servers to better handle incoming traffic. * Improved stability and reduced timeout occurrences across application tier. **Preventive Actions/Ongoing Improvements:** * Continue horizontal scaling of gateway and application components based on traffic patterns. * Periodic review and optimization of network routing and configurations. * Ongoing infrastructure upgrades to significantly improve overall throughput and resiliency. **Resolution Summary \(Post April 11th, 2026\):** Over the weekend, capacity across key network components was further enhanced after confirming congestion-related packet drops. Additionally, required registration updates and configuration changes were implemented to optimize traffic handling and improve request reliability across the network. Following these changes, system stability improved significantly, with no widespread timeout patterns observed.

**Incident Summary:** On March 16th, 2026, clients in the Europe PRD \(FRA SL1, CDG SL1, UK SL1, SL2, SL3\) region experienced a service disruption impacting Cornerstone, resulting in login failures and access issues. **Impact:** Users attempting to access Cornerstone were unable to log in and encountered 504 Gateway Timeout errors. The issue affected authentication flows and prevented successful access to the platform during the incident period. **Root Cause Analysis \(RCA\):** The disruption was caused by latency in the authentication service responsible for handling login requests. Cornerstone relies on a centralized authentication mechanism, and during the incident, authentication requests experienced delays, leading to timeouts at the gateway layer. This resulted in failed login attempts and temporary service unavailability. **Resolution:** Upon identification, the engineering team took corrective actions to stabilize the authentication flow and restore normal response times. System performance was closely monitored, and once latency was reduced, login functionality was fully restored across the impacted environments. **Preventive Measures:** The following measures have been implemented to reduce the likelihood of recurrence: * Enhanced monitoring and alerting for authentication service latency * Reviewed and adjusted timeout thresholds to improve resilience in the request flow * Strengthened capacity planning for authentication related services

**Issue Summary:** On March 4, 2026, users experienced intermittent latency while accessing the platform hosted in the FRA PRD SL1 environment. During this period, some requests took longer than expected to process, resulting in degraded performance across the application. The issue impacted multiple Production environments but did not result in complete service unavailability. **Root Cause:** The latency was caused by stability issues within an underlying authentication service responsible for user authentication and session management. Increased authentication activity, partly driven by the enablement of global SSO capabilities for multiple customers, placed additional load on the service. This resulted in intermittent service instability and elevated request processing times across the affected environments. **Corrective Action:** Cornerstone Engineering teams implemented the following actions to restore service stability: * Increased authentication service capacity to reduce immediate load and stabilize the authentication infrastructure. * Conducted a detailed review of the service implementation to identify inefficiencies. * Optimized the service by removing unnecessary code references and improving memory management. Following these actions, the authentication service stabilized and platform latency returned to normal levels. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone has taken the following actions: * Increased authentication service capacity to better support growing authentication demand. * Implemented code and memory management optimizations to improve service stability and efficiency. * Introduced enhanced monitoring to provide greater visibility into authentication service performance and detect anomalies earlier.

**Issue Summary:** Between February 26 and March 12, 2026, users experienced intermittent latency while accessing the platform hosted in the US SL1, SL2, SL3, and SL5 environments. During this period, some requests took longer than expected to process, resulting in degraded performance across the application. The issue impacted multiple swimlanes but did not result in complete service unavailability. **Root Cause:** The latency was caused by stability issues within an underlying authentication service responsible for user authentication and session management. Increased authentication activity, partly driven by the enablement of global SSO capabilities for multiple customers, placed additional load on the service. This resulted in intermittent service instability and elevated request processing times across the affected environments. **Corrective Action:** Cornerstone Engineering teams implemented the following actions to restore service stability: * Increased authentication service capacity to reduce immediate load and stabilize the authentication infrastructure. * Conducted a detailed review of the service implementation to identify inefficiencies. * Optimized the service by removing unnecessary code references and improving memory management. Following these actions, the authentication service stabilized and platform latency returned to normal levels. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone has taken the following actions: * Increased authentication service capacity to better support growing authentication demand. * Implemented code and memory management optimizations to improve service stability and efficiency. * Introduced enhanced monitoring to provide greater visibility into authentication service performance and detect anomalies earlier.

**Issue Summary:** On March 3, 2026, users experienced intermittent latency while navigating the portal hosted in the FRA SL1 environment. During the incident window, some users encountered slow responses and occasional service unavailable errors while accessing the portal. **Root Cause:** The incident was caused by multiple long-running database sessions executing the same query concurrently. The increased database resource utilization degraded database performance, which propagated latency to the application layer and impacted portal responsiveness. **Corrective Action:** Cornerstone Engineering teams terminated the long-running database sessions causing resource contention and updated database statistics to improve query optimization and execution efficiency. Database replication settings were temporarily adjusted to reduce commit latency while the environment stabilized. System performance was monitored to confirm that database operations and portal access returned to normal. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone is taking the following actions: * Reviewing and optimizing the identified query to prevent prolonged execution. * Enhancing monitoring to proactively detect long-running database sessions and enable faster intervention.

**Issue Summary:** On March 2, 2026, clients hosted in the US SL2 Production environment experienced a temporary service disruption affecting the Career Site. During the incident window, users encountered intermittent slowness and brief periods of unavailability while accessing the Career Site. **Root Cause:** The incident was caused by an unexpected surge in traffic to the Career Site during off-hours. The elevated request volume temporarily exceeded available application capacity, resulting in service instability. **Corrective Action:** Cornerstone Engineering teams increased application capacity to stabilize performance and restore normal service levels. System performance was monitored following the adjustment to confirm that service stability had been fully restored. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone is implementing the following improvements: * Enhancing monitoring thresholds to detect sudden traffic spikes earlier and enable faster response. * Reviewing and adjusting capacity planning to better accommodate unexpected increases in traffic.

**Issue Summary:** On February 28, 2026, users experienced intermittent service availability while accessing the portal hosted in the US SL1 Production environment. During the incident window, some users encountered intermittent access issues when navigating the portal. **Root Cause:** The incident was caused by a configuration limit being reached within the underlying infrastructure service. Once the limit was reached, normal traffic handling was temporarily impacted, which resulted in intermittent portal accessibility issues. **Corrective Action:** Cornerstone Engineering teams identified the configuration limit and implemented adjustments to restore normal service operation. The environment was monitored following the change to confirm stability and ensure that portal access had returned to normal. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone has taken the following actions: * Increased the relevant infrastructure service limit to support higher capacity and prevent similar constraints from affecting portal accessibility in the future.

**Issue Summary:** Between February 26 and March 12, 2026, users experienced intermittent latency while accessing the platform hosted in the US SL1, SL2, SL3, and SL5 environments. During this period, some requests took longer than expected to process, resulting in degraded performance across the application. The issue impacted multiple swimlanes but did not result in complete service unavailability. **Root Cause:** The latency was caused by stability issues within an underlying authentication service responsible for user authentication and session management. Increased authentication activity, partly driven by the enablement of global SSO capabilities for multiple customers, placed additional load on the service. This resulted in intermittent service instability and elevated request processing times across the affected environments. **Corrective Action:** Cornerstone Engineering teams implemented the following actions to restore service stability: * Increased authentication service capacity to reduce immediate load and stabilize the authentication infrastructure. * Conducted a detailed review of the service implementation to identify inefficiencies. * Optimized the service by removing unnecessary code references and improving memory management. Following these actions, the authentication service stabilized and platform latency returned to normal levels. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone has taken the following actions: * Increased authentication service capacity to better support growing authentication demand. * Implemented code and memory management optimizations to improve service stability and efficiency. * Introduced enhanced monitoring to provide greater visibility into authentication service performance and detect anomalies earlier.

**Issue Summary:** On February 24, 2026, elevated latency was observed in the EU region, resulting in degraded platform performance. During the affected period, users experienced slower response times while accessing the platform, particularly for operations within the Learning Experience Platform \(LXP\). **Root Cause:** The incident was caused by performance degradation in an interdependent backend service. The slowdown in this service propagated to dependent platform components, resulting in increased latency and delayed processing of LXP operations in the EU region. **Corrective Action:** As an immediate mitigation, Cloud Operations collaborated with the development team to implement temporary measures to reduce the latency impact and stabilize platform performance. Engineering teams have also prepared a permanent code fix to address the underlying issue. **Preventive Measures:** To reduce the likelihood of a similar incident in the future, the following fix is being implemented: * Improved the service logic to better handle certain user validation scenarios, preventing unnecessary retries that previously contributed to increased latency.

**Issue Summary:** On February 23, 2026, increased latency was observed across all U.S. swimlane environments due to an issue with an internal service that depends on a third-party database service \(ArangoDB\). The incident affected Galaxy customers using LXP and LMS, including environments running LXP and LMS together, or LXP alone. During the affected period, users experienced slower page load times and delayed system responses. **Root Cause:** The incident was caused by a disruption in the third-party ArangoDB service used by an internal platform component. The dependency caused the internal service to experience latency, which propagated to all US swimlane environments and resulted in broader platform responsiveness degradation. **Corrective Action:** Engineering teams collaborated with the ArangoDB vendor to identify and address the service disruption. Once the vendor restored normal operation of the database service, the dependent internal services recovered and system performance returned to expected levels. **Preventive Measures:** To reduce the likelihood of a similar incident in the future, the following improvement is being implemented: * Implement Circuit Breaker: Introducing circuit breaker within internal services to prevent dependency failures from propagating and causing broader platform latency.

**Incident Summary:** On February 17th, 2026, users were unable to log in to the portal hosted in the CDG SL1 \(AWS\) environment. During the impact window users encountered Service Unavailable errors. **Impact:** During the incident window, portal login functionality was temporarily unavailable. Users attempting to access the portal received service unavailable errors. The issue was limited to the affected environment. **Root Cause Analysis \(RCA\):** The disruption was caused by multiple long running database sessions executing the same query concurrently. The increased database resource utilization degraded overall performance, which in turn caused the application layer to return errors. **Resolution:** Upon identification of the issue, the engineering team: * Terminated the long-running database sessions causing resource contention and updated database statistics to improve query optimization and execution efficiency. * Temporarily adjusted database replication settings to reduce commit latency during stabilization. * Closely monitored system resource utilization to confirm normalization. Following these actions, database performance stabilized and portal access was fully restored. **Preventive Measures:** To reduce the likelihood of recurrence, the following measures are being implemented: * Reviewing and optimizing the identified query to prevent prolonged execution * Enhancing monitoring to proactively detect long running database sessions

**Incident Summary:** On February 9th, 2026, clients hosted in the US Production environments \(SL1, SL2, SL3, and SL5\) experienced a temporary service disruption affecting the Career Site. **Impact:** During the incident window, users experienced intermittent slowness and brief periods of unavailability while accessing the Career Site. The impact was limited to elevated traffic conditions during off-peak hours. **Root Cause Analysis \(RCA\):** The disruption was caused by an unexpected surge in traffic to the Career Site during off-hours. The elevated request volume temporarily exceeded the available application capacity, which resulted in service instability. **Resolution:** Upon identification of the issue, Cornerstone engineers promptly increased application capacity to stabilize performance and restore normal service levels. Platform performance was continuously monitored to ensure full recovery and sustained stability. **Preventive Measures:** To reduce the likelihood of recurrence, Cornerstone is implementing the following improvements: Adjusting auto-scaling schedules to better accommodate traffic patterns during off-hours Enhancing monitoring thresholds to proactively respond to sudden traffic spikes Reviewing capacity planning assumptions to ensure sufficient headroom during peak and off-peak periods