House of Control incident
[Complete Control] - Service Disruption
House of Control experienced a critical incident on July 8, 2022 affecting Complete Control, Region - Norway and Complete Control, Region - Denmark, lasting 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 08, 2022, 01:16 PM UTC
We are currently experiencing a service disruption causing new logins to fail. Our Operations team is working to identify the root cause and implement a solution. We will send an additional update in 30 minutes or less.
- investigating Jul 08, 2022, 01:28 PM UTC
We are continuing to investigate this issue.
- resolved Jul 08, 2022, 01:29 PM UTC
This incident has been resolved.
- postmortem Jul 12, 2022, 07:07 AM UTC
During regular maintenance, several folders were cleared to free up extra disk space for the operation during the summer holiday \(fellesferien\). It turned out that one token file that was needed for communication between two of the subsystems was lost during the cleanup. As a result, end users were unable to log in to Complete Control. A backup had been taken before the actual clean-up started. The incident was resolved by rolling back the token file from the backup. ### Timeline * 14:05: The directories were backed up. * 14:36: The contents of these directories were deleted * 14:36: The incident started * 15:16: A notification about the incident published on statuspage * 15:16: The file with the Backoffice token was restored from the backup. This resolved the incident * 15:29: A resolved message was published on statuspage ### Measures A fix will be implemented in the application. The token file is moved to to another location where it will not be subject to cleanup operations.