Intermittent Latency and Errors Across Application - US Swimlanes

Update timeline

1. investigating Jul 14, 2025, 09:11 PM UTC

   Some customers on a US swimlane may experience latency and errors intermittently across the application. We are actively investigating the issue as our top priority. We will share regular updates as they become available.

2. investigating Jul 14, 2025, 10:40 PM UTC

   Our engineering team is continuing to investigate the root issue. We will update here once the problem has been identified. This remains a top priority for our team.

3. identified Jul 14, 2025, 11:49 PM UTC

   The issue has been identified and our engineering team is working on developing a fix. The issue has been mitigated for now. We will update here once a resolution is in place. This remains a top priority for our team.

4. resolved Jul 15, 2025, 01:28 AM UTC

   The CSOD Technology Team observed a service disruption on the US swimlane. Users were intermittently encountering a 504 error when logging into the portal. The issue now stands resolved.

5. postmortem Jul 28, 2025, 03:57 PM UTC

   **Incident Summary:** On July 14, 2025, an abnormal spike in incoming traffic was detected, triggering automated protection mechanisms and temporarily disrupting normal request processing. As a result, customers hosted in the US Production, Stage, and Pilot environments experienced intermittent portal access errors and delays. **Root Cause:** The disruptions were caused by protective systems engaging to assess and filter unusual traffic patterns. The traffic spike was identified as non-standard and potentially automated. While the exact source could not be conclusively determined, the pattern was deemed anomalous and inconsistent with normal user activity. Existing protection measures mitigated much of the impact, though tuning adjustments were required to improve detection and response behavior. **Corrective Action:** Stability was restored by refining the traffic filtering logic and validating the legitimacy of incoming requests. Monitoring was elevated, and configuration updates were applied to return systems to normal operating levels. **Preventive Measures:** To reduce the likelihood of similar incidents, Cornerstone is taking the following actions: * Reviewing and refining protection rules to better manage anomalous traffic without disrupting legitimate users * Evaluating and adjusting filtering modes to achieve an optimal balance between security and accessibility * Enhancing anomaly detection thresholds and response logic to improve system resilience under unexpected load conditions