Cornerstone incident
Intermittent timeout EU PRD - 25% of Customers Impacted
Cornerstone experienced a notice incident on August 30, 2025 affecting Uptime and Response Time, lasting 9h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 30, 2025, 09:20 AM UTC
The CSOD Technology Team observed a intermittent timeouts in the EU PRD. The team is currently working and monitoring to fix the issue completely. Start time: 12:50 am PDT
- investigating Aug 30, 2025, 12:53 PM UTC
The team is actively investigating the issue to determine the solution. We will continue to share updates as we learn more.
- investigating Aug 30, 2025, 01:43 PM UTC
We are continuing to investigate this issue.
- investigating Aug 30, 2025, 05:42 PM UTC
The team is actively working on it. We will continue to share updates as we learn more.
- resolved Aug 30, 2025, 06:45 PM UTC
The incident has been resolved at 10:30 AM PDT.
- postmortem Nov 06, 2025, 07:46 AM UTC
**Incident Summary:** On August 30th, 2025, clients hosted in the EU Production environment experienced intermittent latency while accessing their portals between 12:58 AM and 1:02 AM PDT. **Root Cause Analysis \(RCA\):** The issue was caused by a long-running database recovery operation that temporarily increased system load and resource utilization. This led to short periods of blocking and query timeouts, impacting portal responsiveness for some clients. **Resolution:** Engineering teams optimized the recovery process and performed corrective actions to restore normal performance. The environment has since stabilized, and portal access has returned to standard operating levels. **Preventive Measures:** * Schedule long-running database recovery or maintenance operations during planned maintenance windows to minimize impact. * Enhance monitoring and alerting to proactively detect and address similar performance conditions in the future.