TechnologyOne incident
Scheduled Maintenance - [Mainline Upgrade] - ANZ Region / 2023B & 2024A Release
TechnologyOne experienced a notice incident on July 1, 2024, lasting 21h 53m. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Jul 01, 2024, 03:16 AM UTC
Please note that the scheduled Mainline upgrade on the weekend of 29th of June (Saturday) was partially completed. This means that a number of our customer environments were not upgraded as per the schedule. For customers who still have not received the upgrade, we will run the upgrade again at 8PM AEST today, 1st July 2024. Our teams are conducting thorough investigation to determine the cause which prevented the complete roll out of the upgrades.
- identified Jul 01, 2024, 04:07 AM UTC
We still have a small number of mainline environments which haven't been upgraded over the weekend unfortunately. You can confirm if you are affected by checking the environment snapshot number. For 2023B this is 635 For 2024A this is 384 If you are not currently on this release in Mainline, we will be upgrading these environments again at 8PM tonight.
- monitoring Jul 01, 2024, 11:08 PM UTC
We have successfully rolled out the mainline upgrades to all customers last night (1 July 2024, 8PM AEST) We will continue to monitor this for the next 2 hours before closing this out. Thank you
- resolved Jul 02, 2024, 01:10 AM UTC
The incident has been resolved. We are currently investigating the cause and will provide an update once our investigation is complete.
- postmortem Jul 02, 2024, 01:11 AM UTC
**Issue Summary** The Mainline upgrade for 2023B \(635\) and 2024A \(384\) was scheduled to be deployed to all mainline customers on the 29th of June 2024. For a subset of customer datasets, the upgrade failed to deploy, resulting in the need to reschedule these for upgrade the following day \(30th June 2024\). **Root Cause** * Initially a service disruption caused the upgrade process to fail to start for some customer data sets and complete for other customer data sets. * A subsequent degradation was found in the service causing slow response times which lead to incomplete upgrades. **Corrective Actions** 1. We rescheduled the upgrades for the affected customer datasets as necessary. **Preventive Actions** 1. Research options to automate a retry for the deployment service and implement. 2. Improve the queries that caused slow response times.