Latency on FRA SL1 Production

Update timeline

1. resolved Mar 03, 2026, 12:11 PM UTC

   On 3/3/2026 between 1:00 AM and 2:30 AM PST we experienced latency in FRA SL1. During this time portals may have experienced issues accessing or using the platform.

2. postmortem Mar 14, 2026, 08:31 PM UTC

   **Issue Summary:** On March 3, 2026, users experienced intermittent latency while navigating the portal hosted in the FRA SL1 environment. During the incident window, some users encountered slow responses and occasional service unavailable errors while accessing the portal. **Root Cause:** The incident was caused by multiple long-running database sessions executing the same query concurrently. The increased database resource utilization degraded database performance, which propagated latency to the application layer and impacted portal responsiveness. **Corrective Action:** Cornerstone Engineering teams terminated the long-running database sessions causing resource contention and updated database statistics to improve query optimization and execution efficiency. Database replication settings were temporarily adjusted to reduce commit latency while the environment stabilized. System performance was monitored to confirm that database operations and portal access returned to normal. **Preventive Measures:** To reduce the risk of recurrence, Cornerstone is taking the following actions: * Reviewing and optimizing the identified query to prevent prolonged execution. * Enhancing monitoring to proactively detect long-running database sessions and enable faster intervention.