Beyond Identity incident

Authentication service failed to restart

Critical Resolved View vendor source →

Beyond Identity experienced a critical incident on October 26, 2020, lasting —. The incident has been resolved; the full update timeline is below.

Started
Oct 26, 2020, 08:27 PM UTC
Resolved
Oct 22, 2020, 10:30 AM UTC
Duration
—
Detected by Pingoru
Oct 26, 2020, 08:27 PM UTC

Update timeline

  1. resolved Oct 26, 2020, 08:27 PM UTC

    While performing a routine deployment of a new version to the production, an anomaly occurred, causing the authentication service to fail to start up.

  2. postmortem Oct 26, 2020, 08:27 PM UTC

    **Leadup** While performing a routine deployment of a new version to production, an anomaly occurred, causing the authentication service to fail to start up. **Fault** A database migration failed, causing subsequent database migration tasks to not execute. **Detection** QA immediately ran a test after the deployment to log out and log back into production using Beyond Identity Authenticator, and they observed the issue. **Root cause** Failed database migration left the database in a state not expected by the cloud services. **Mitigation and resolution** The Site Reliability Team escalated to the cloud engineering team, who identified the root cause and prepared a manual fix to the content in the database. After implementing the data repair, the SRE team re-executed the routine deployment successfully, restoring all services back to their normal state.