Cornerstone experienced a notice incident on July 28, 2025, lasting 1h 21m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Jul 28, 2025, 08:46 PM UTC
Customers on LAX SL4 may encounters errors and issues trying to log into Production and Pilot. We are actively investigating the issue as our top priority. We will share regular updates as they become available.
- monitoring Jul 28, 2025, 09:01 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 28, 2025, 10:08 PM UTC
The CSOD Technology Team observed a service disruption on this swimlane. The problem began at 1:25 PM Pacific Time and service was restored at 1:45 PM Pacific Time. During this time, clients with portals on this swimlane were not able to access the application. After a period of monitoring we are marking this incident resolved.
- postmortem Aug 01, 2025, 10:05 PM UTC
Incident Summary: On July 28, 2025, clients reported an issue in the LAX PRD and PIL SL4 environment affecting access to the platform. Impact: Clients were unable to log in to the portal during the incident window Root Cause Analysis \(RCA\): The disruption was caused by a failure in a backend database service, which broke application connectivity and interrupted login functionality. The underlying issue was traced to contention and buildup of audit logs in the database. This led to resource contention within the database engine, which degraded performance and eventually caused the service to become unresponsive. The failure was further compounded by a delay in the automated recovery mechanism, which extended the outage duration. Corrective Actions: The engineering team took immediate steps to restore functionality, including clearing impacted service components, restoring access, and validating system stability. Full access to the platform was reinstated following these actions. Preventive Measures: To prevent recurrence, the following measures have been implemented: o System housekeeping routines have been reviewed and reinforced to ensure consistent execution. o Failover mechanisms are being enhanced to improve responsiveness and recovery time in the event of future disruptions.