Cornerstone incident
Intermittent timeout on US SL5 - 25% of Customers Impacted
Cornerstone experienced a minor incident on October 30, 2025 affecting Response Time, lasting 1h 45m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 30, 2025, 04:59 PM UTC
The CSOD Technology Team observed a intermittent timeouts in the US SL5. The team is currently investigating the issue. Start time: 06:00 am PDT.
- monitoring Oct 30, 2025, 05:11 PM UTC
The incident has been resolved at 9:13 AM PDT.
- resolved Oct 30, 2025, 06:44 PM UTC
This incident has been resolved.
- postmortem Nov 06, 2025, 07:56 AM UTC
**Incident Summary:** On October 30th, 2025, clients hosted in the US Production SL5 environment may have experienced intermittent latency and timeout errors while accessing their portals between 6:00 AM and 9:13 AM PST. **Root Cause Analysis \(RCA\):** The latency was caused by temporary blocking within the database system, resulting in slower response times and timeout alerts. The issue was linked to synchronization delays and high system resource utilization, which caused intermittent performance degradation during the observed timeframes. **Resolution:** Engineering teams performed corrective actions, including system optimization and a controlled restart, which stabilized performance. The environment has remained stable since 10/30 at 9:13 AM PST, with no further latency or timeout alerts observed. **Preventive Measures:** * Enhance system monitoring to detect early signs of blocking or synchronization delays. * Review and optimize database performance parameters to reduce the likelihood of similar latency events. * Schedule proactive maintenance during low-usage windows to ensure continued stability.