[Complete Control] Service disruption Complete Control, Norway

Update timeline

1. resolved May 02, 2022, 09:52 AM UTC

   Some users may be experiencing issues when logging in to Complete Control Norway. The disruptions started at about 10:25, and lasted until about 10:52. The issue is now solved. Our operations team is currently investigating to identify the root cause. We will update this page as soon as we have more details about the incident. We apologize for any inconvenience this may have caused to your use of our services.

2. postmortem May 06, 2022, 06:02 AM UTC

   ## Incident interval * 2022-05-02 09:54-2022-05-02 09:56, on server NO-01, 00:02 hours downtime * 2022-05-02 10:26-2022-05-02 10:42, on server NO-02, 00:16 hours downtime * 2022-05-02 10:50-2022-05-02 10:52, on server NO-01, 00:02 hours downtime * 2022-05-02 19:35-2022-05-02 21:55, on server NO-01, 01:20 hours downtime ## Cause An error made during deployment of a new release caused the Complete Control application to crash with a HTTP 500 error. A component not belonging to the release was accidentally included in the deployment. This was the direct cause of the incident. ## Resolution The issue was resolved by manually removing the component that was included by mistake. ## Measures * Limit the permissions to execute specific commands in the production environment to the Operations team. * Improve the quality assurance of the contents of critical parts of the release. * Review of the release process to ensure the necessary competence in all steps, and clear segregation of duties during releases.