postmortem Mar 25, 2026, 06:35 PM UTC
UA Google User Access Interruption Problem Impact Analysis Event Occurrence: Monday March 9th @ 1:24 PM # Background UA Google Workspace is the primary cloud collaboration service provided by UA OIT to all UA users. At its most basic functions, it provides email inboxes to all users, as well as access to tools such as Docs, Drive, Sheets, and more. # Break Down of the Problem On Monday March 9th at approximately 1:00 PM, OIT PAWs began to conduct a planned activity within UA Workspace that applied archive license status to a predetermined set of accounts that exceeded the recently established account retention policies for cloud services such as Google. At approximately 1:24 PM, PAWS was notified by UAF NTS via Slack that several users had reached out to the NTS Service Desk, indicating that their access to UA Workspace has been terminated. OIT PAWs began investigating the matter alongside NTS Technicians. At approximately 2:10 PM, the account archival process was halted. NTS SD Technicians were able to assist with immediate account extensions for users who reached out, alongside an un-archival of their user accounts in Workspace. A rollback of the process was initiated, and completed by approximately 3:30 PM. # Target State / Goal All UA users with active status with UA should retain access to their UA Google Workspace accounts. # Root Cause Analysis Investigations with NTS SD Technicians pointed to a common cause: incorrectly extended accounts. The affected accounts all had extended AD statuses, but had mistakenly not been assigned one of the roles used to identify active user accounts. For the most part, this pointed to alumni users that were missing the \`UAF\_Patrons\` role, but also included at least one service account missing a similar role.Of the ~8000/100 000 records processed during this time, we were able to identify approximately 13 user accounts that may have been impacted by this. # Develop Countermeasures NTS SD Technicians began immediate outreach to members of the UA AD team to capture a list of users that may have inappropriately extended accounts \(e.g. missing role types\), in an effort to remediate misidentifying users. In addition, we would proactively remove any active AD user from archival processing in favor of manual review to avoid similar issues with incorrect account statuses and roles. # Implementation of Countermeasures User filtering criteria were updated immediately. Additionally, this activity was put through UA CAB later the same week for increased visibility. # Follow Up / Review This activity was completed later in the week, on March 12, with no further incident.
