Versaic: Login failure - unexpected server error

Update timeline

1. monitoring Jun 23, 2025, 03:17 PM UTC

   A fix has been implemented and we are monitoring the results.

2. resolved Jun 23, 2025, 05:16 PM UTC

   This incident has been resolved.

3. postmortem Jun 26, 2025, 09:12 PM UTC

   ## Summary On Monday, June 23, 2025, at approximately 08:03 Mountain Time \(MT\), a disruption occurred during a routine service update to the Versaic application platform. The issue affected user access to both the Client and Admin portals, resulting in login failures and an error message display. The incident was resolved within approximately one hour, with all services fully restored by 09:05 MT. ## Impact From 08:03 MT to 09:05 MT, users attempting to log in to the Versaic Client or Admin portals were unable to access the system. Instead of reaching the application’s dashboard, users encountered a message indicating: “An unexpected server error occurred.” During this period: * No user data was lost, corrupted, or compromised. * The issue was isolated to the login functionality; the underlying data and platform services remained secure and unaffected. ## Root Cause The interruption was caused by an unexpected failure during a database update as part of a scheduled deployment. Although automated system checks are designed to detect such issues before any changes are released to the live environment, this particular failure mode was not identified in time. As a result, the new application version was released, causing the login issues experienced by users. Our technical teams were immediately engaged upon alert and began diagnosing the issue. Within approximately 20 minutes, the primary cause was identified, and remediation actions were promptly executed. Full functionality was restored by 09:05 MT. ## Future Mitigation To help prevent similar incidents in the future, we are taking the following steps: * Enhancing failure detection mechanisms within our deployment processes. * Introducing additional validation checkpoints before changes are made live. ## Timeline of Events * 08:03 MT - Initial alert received * 08:20 MT - Major contributing cause identified * 08:25 MT - Notable action taken * 09:25 MT - Incident resolved; systems fully operational