Cornerstone incident

Intermittent Errors - US SL3

Notice Resolved View vendor source →

Cornerstone experienced a notice incident on December 12, 2025, lasting —. The incident has been resolved; the full update timeline is below.

Started
Dec 12, 2025, 11:37 PM UTC
Resolved
Nov 16, 2025, 09:30 PM UTC
Duration
Detected by Pingoru
Dec 12, 2025, 11:37 PM UTC

Update timeline

  1. resolved Dec 12, 2025, 11:37 PM UTC

    Some customers hosted on US SL3 may have experienced intermittent errors when accessing the application. The issue occurred on November 16 between 4:28 PM PT and 5:15 PM PT and has been resolved.

  2. postmortem Jan 06, 2026, 05:16 PM UTC

    **Incident Summary:** On November 16th, 2025, Cornerstone engineers were alerted to performance degradation and database-related alerts impacting clients on US SL3 production environment. During this period, certain background processing experienced delays, which contributed to increased system load and temporary performance impact for affected clients. **Root Cause Analysis \(RCA\):** The issue was caused by long-running database processes related to global filter processing, which held open transactions for extended periods. These transactions led to resource contention causing performance degradation. **Corrective Action:** * Cornerstone engineering teams investigated the issue and identified the long-running processes contributing to the contention. * Impacted database sessions were terminated, which immediately released resources and restored normal system behavior. * As an interim safeguard, automated controls were updated to proactively terminate the affected processes if utilization crosses defined thresholds. * System stability and performance were validated following remediation. **Preventive Measures:** To reduce the likelihood of recurrence, Cornerstone is taking the following actions: * Reviewing and implementing database query optimizations to improve efficiency of global filter processing. * Enabling optimized processing paths through configuration where applicable. * Enhancing monitoring to detect long-running transactions earlier and trigger proactive remediation.