injixo incident

Virtual ShiftCenter shows logon failure and can not be used.

Critical Resolved View vendor source →

injixo experienced a critical incident on August 13, 2022, lasting 3h 56m. The incident has been resolved; the full update timeline is below.

Started
Aug 13, 2022, 08:24 AM UTC
Resolved
Aug 13, 2022, 12:20 PM UTC
Duration
3h 56m
Detected by Pingoru
Aug 13, 2022, 08:24 AM UTC

Update timeline

  1. investigating Aug 13, 2022, 08:24 AM UTC

    We are currently investigating this issue.

  2. resolved Aug 13, 2022, 12:20 PM UTC

    This incident has been resolved.

  3. postmortem Sep 05, 2022, 09:47 AM UTC

    **Summary** On Friday 12.08.2022 around 22:00 CET we got informed by our customer experience team that customers were unable to work with Plan/ShiftCenter. 30 minutes after first report we managed to rollback our latest changes. Unexpectedly, however, this did not solve the problem and further investigations became necessary. 12 hours after first report we involved our cloud-ops team to downgrade the machine to a previous version. The machine rollback resolved the issue instantly. **Fault** Any attempt to open a ShiftCenter leads to the error message, that the current user can’t be logged in. **Impact** Starting 12.08.2022 22:00 CET until 13.08.2022 14:20 CET all customer using Plan/ShiftCenter were unable to use it. **Detection** The incident was detected by our customers and got reported via customer experience team to development. **Response** We responded to customers on 13.08.2022 10:00 CET about a major outage of Plan/ShiftCenter at [status.injixo.com](http://status.injixo.com/) **Recovery** We rolled back all code changes made by our team first. As this did not help at all we rolled back all changes made on the machine. This resolved the issue. **Root cause** Microsoft Windows patch KB5014022 changed the login behaviour for Microsoft Windows. This change requires an updated component. **Corrective actions** We developed a multi-user test for our ShiftCenter rollout. We apologise for the service disruption.