ACA Global incident
Update to Jira (Track-It) causing error in worklogs
ACA Global experienced a major incident on August 9, 2021, lasting 29d 16h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Aug 09, 2021, 08:34 PM UTC
Due to the essential CVE upgrade 8.13.9 performed today we discovered a critical bug in the worklogs on our Jira / Trackit. These worklogs have shifted one working day. For the time being please take caution in analyzing and evaluating time reports.
- investigating Aug 09, 2021, 08:36 PM UTC
We have created JIR-1936 for internal followup. We will post updates when available.
- investigating Aug 10, 2021, 07:23 AM UTC
To prevent further data corruption in the worklogs to occur, we will be disabling the worklogs for the time being. Meaning you will not be able to create new worklog entries until further notice.
- identified Aug 10, 2021, 09:01 AM UTC
We discovered that with the upgrade from 8.13.3 to 8.13.9 last night a problem occured in the timezone interpretation from log entries from 06.03.2021 till last night. We are able to reproduce the problem om accept and are performing a downgrade to test the behavior and contacted Atlassian for additional support. Please stand by for updates throughout the day.
- identified Aug 10, 2021, 11:38 AM UTC
We found the root cause of the issue, related to the version installed in March and the effect of the update yesterday. We are currently updating Jira Acceptance to work on a data fix for all time logged and planned. We will reopen the ability to log and plan your work now. But please do NOT update or re-create your wrong work or planned entries of the last days yet, this will be fixed using a mass script in the coming days!
- identified Aug 11, 2021, 03:58 PM UTC
We have succesfully updated Jira Acceptance Tempo Timesheet Worklogs and Planning elements using a SQL script to change to the correct Timezone. After multiple validations throughout the day we have a GO to perform the script in Production this evening. After the script ran we will continue the investigation and resolution to fix the Submitted TimeSheet Approval status for the months March till July.
- monitoring Aug 11, 2021, 06:31 PM UTC
The script to fix your worklog entries and planning entries on production has been successfully implemented and Jira has been restarted. You can now resume to generate reports and planning acitivities. If you do notice strange behaviour, please create a support issue with all possible details on support.aca-it.be. We will keep this thread open until we implemented a solution for the Timesheet Approval status.
- resolved Sep 08, 2021, 12:40 PM UTC
All timesheets have been approved and validated.