TechnologyOne Outage History

Started Feb 03, 2025, 12:38 AM UTC· resolved Feb 03, 2025, 10:07 PM UTC

1. investigating Feb 03, 2025, 12:38 AM UTC

   We are investigating an issue impacting service for ANZ Region for all Release. A subset of customers in ANZ Region for all Release are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. investigating Feb 03, 2025, 01:39 AM UTC

   Our team is continuing to investigate an issue impacting service for ANZ Region for All Releases. A subset of customers in ANZ Region for All Releases are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

3. monitoring Feb 03, 2025, 02:02 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region and all Releases. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

4. monitoring Feb 03, 2025, 03:08 AM UTC

   Monitoring Update: We have observed the majority of Production environments have been restored and working again. For non-Production environments we continue to apply the fix. Next update expected in 60 mins.

5. monitoring Feb 03, 2025, 04:18 AM UTC

   The Production environments have been restored and working again. For non-Production environments we continue to apply the fix.

6. monitoring Feb 03, 2025, 05:23 AM UTC

   The Production environments have been restored and working again. The issue is intermittent in non-production environment, and we are working to stabilize Next update expected in 2 hours.

7. monitoring Feb 03, 2025, 07:10 AM UTC

   We are continuing to work on stabilising the non-production environments. Monitoring indicates the issues in non-production environments are intermittent. Monitoring confirms all Production environments have been restored and remain stable. Next update expected in 2 hours.

8. monitoring Feb 03, 2025, 09:06 AM UTC

   A fix has been implemented for the impacted non-production environments in the ANZ Region. Monitoring confirms all Production environments have been restored and remain stable. We will continue to monitor Production and Non-Production environments to ensure no further issues. Next update will be provided by 9am AEST 4/2/2025.

9. resolved Feb 03, 2025, 10:07 PM UTC

   After considerable monitoring this incident is now resolved. We apologise for how you and your business may have been affected by this incident.

10. postmortem Feb 13, 2025, 10:27 PM UTC

    **Issue Summary** On 3 February 2025 a partial failure in the network occurred at 10.00am AEST. This impacted a subset of customers who then experienced a range of 5XX errors, and many users were unable to login. This disruption was between 10 to 15 minutes for the majority of customers, impacted on Production however the recovery continued until 2pm AEST for selected customers. The TechnologyOne teams first priority is to stabilise the Production environments, followed by recovery of non-production environments. TechnologyOne teams developed mitigations to recover from any re-occurrences of the issue until preventative actions could be established and implemented. TechnologyOne teams continued to work and analyse the root cause and develop actions through to 10/2/2025 Preventative actions began being applied progressively across regions from 2pm on 10/2/2025. **Root Cause** A partial failure on managed DNS caused network instability leading to intermittent access to application infrastructure. **Corrective Actions** Infrastructure was systematically reviewed and reset to recover from the hung state. Developed mitigations to recover for any re-occurrences. The managed DNS was recovered by our upstream vendor. **Preventative Actions** Implemented configuration changes in managed DNS based on recommendations from our upstream provider who reviewed the workload patterns to provide additional resilience to the service.

Read the full incident report →

Started Jan 21, 2025, 09:39 AM UTC· resolved Jan 21, 2025, 07:08 PM UTC

1. investigating Jan 21, 2025, 09:39 AM UTC

   Please be advised that we are aware of Resource Booker performance issues and the team are actively working towards a resolution.

2. identified Jan 21, 2025, 09:49 AM UTC

   The team have identified the issue and are working on a solution.

3. identified Jan 21, 2025, 10:53 AM UTC

   Progress is being made in restoring services. We continue to deploy our fix and will update once service has been fully restored

4. identified Jan 21, 2025, 01:14 PM UTC

   We continue to make good progress in restoring services. Work is still ongoing to fully restore services.

5. monitoring Jan 21, 2025, 04:55 PM UTC

   Performance and stability continues to improve. We're now monitoring this issue

6. resolved Jan 21, 2025, 07:08 PM UTC

   Our engineers are now happy with the performance and stability of RB and PB.

7. postmortem Feb 12, 2025, 06:57 AM UTC

   Issue Summary: On Tue, Jan 21, a subset of customers experienced sub-optimal performance with Resource Booker, including slow response times and errors loading admin functions. Root Cause: The system degradation was caused by Resource Booker overloading the system with large volume of data when it searched for pending approvals and found none. This was caused by a bug that was introduced into the software that surfaced in the production environment. Corrective Actions: Temporarily optimised the search services to handle more requests. Preventive Measures: Updated Resource booker to only search for events when necessary and only a subset of the data.

Read the full incident report →

Started Dec 09, 2024, 01:33 PM UTC· resolved Dec 09, 2024, 10:33 PM UTC

1. investigating Dec 09, 2024, 01:33 PM UTC

   We are investigating an issue impacting service for the ANZ Region. A subset of customers in the ANZ Region are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. monitoring Dec 09, 2024, 04:00 PM UTC

   Our team has verified the implementation of a fix is complete. We will monitor the logs for the next few hours to ensure no further customers are impacted.

3. resolved Dec 09, 2024, 10:33 PM UTC

   This incident has been resolved.

4. postmortem Dec 24, 2024, 06:07 AM UTC

   **Issue Summary:** On Monday, December 09, 2024, A subset of customers in the ANZ Region were experiencing sub-optimal performance with connection errors in Ci Anywhere / Ci. ‌ **Root Cause:** The degradation was caused by an upstream service. ‌ **Corrective Actions:** The upstream provider fixed the impaired servers to restore the service. ‌ **Preventive Measures:** Enhance the scalability and performance underload

Read the full incident report →

Started Nov 27, 2024, 03:53 PM UTC· resolved Nov 27, 2024, 06:19 PM UTC

1. investigating Nov 27, 2024, 03:53 PM UTC

   We are investigating an issue impacting service for UK Region. Impact: A subset of customers in UK Region are experiencing sub-optimal performance. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Nov 27, 2024, 04:21 PM UTC

   Team have identified root cause and are currently working towards resolution

3. monitoring Nov 27, 2024, 05:19 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 27, 2024, 06:19 PM UTC

   Following the implementation of the resolution, the errors have ceased, and the service has been fully restored to normal operation.

5. postmortem Dec 18, 2024, 01:13 AM UTC

   **Issue Summary:** On Wednesday, November 27, 2024, at 3:20 PM UK time, a subset of the customers experienced sub-optimal performance using server folders. The TechnologyOne team quickly investigated and found a solution. **Root Cause:** The issue was caused by an outdated entry in the system’s file management. **Corrective Actions:** Removed the outdated entry and cleared the system to fix the problem. ‌ **Preventive Measures:** Implemented a change to automatically detect and remove outdated entries.

Read the full incident report →

Started Nov 27, 2024, 02:52 AM UTC· resolved Nov 28, 2024, 06:07 AM UTC

1. investigating Nov 27, 2024, 02:52 AM UTC

   We are investigating an issue impacting service for 2024A Spatial cloud. A subset of customers for 2024A Spatial Cloud are experiencing sub-optimal performance. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Nov 27, 2024, 03:49 AM UTC

   Our team has identified the issue causing sub-optimal performance and are now working on a fix. We will provide another update in 60 minutes, or sooner if new information becomes available.

3. identified Nov 27, 2024, 04:48 AM UTC

   Further investigation has been undertaken and this issue is impacting a subset of customers on spatial cloud across ALL releases. We have identified the issue on our data layer and continuing to work on remediation. Intramaps appears to be functional however Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance. We will provide a further update in 2 hours, or sooner if new information becomes available.

4. identified Nov 27, 2024, 06:53 AM UTC

   We are continuing to work on remediation. Intramaps continues to be functional. Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance and are likely to present with errors at times that can often be clicked through. We will provide a further update at 9am AEST, or sooner if new information becomes available.

5. identified Nov 27, 2024, 10:25 PM UTC

   A fix is currently being implemented. We expect the fix to be completed at approximately 3pm AEST today. Intramaps continues to be functional. Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance and are likely to present with errors at times that can often be clicked through. We will provide a further update at 1pm AEST, or sooner if new information becomes available.

6. identified Nov 28, 2024, 02:31 AM UTC

   The fix that is currently being implemented remains on track for completion by 3pm AEST today. Intramaps continues to be functional. Map Manager, Tile Engine and Config Manager are continuing to experience degraded performance and are likely to present with errors at times that can often be clicked through. We will provide a further update at 3pm AEST, or sooner if new information becomes available.

7. monitoring Nov 28, 2024, 03:27 AM UTC

   Our team has verified the implementation of a fix is complete for the subset of customers on Spatial Cloud that have been affected by sub-optimal performance. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

8. resolved Nov 28, 2024, 06:07 AM UTC

   After 2 hours monitoring this incident is now resolved. We apologise for how you and your business may have been affected by this incident.

9. postmortem Dec 10, 2024, 06:02 AM UTC

   **Issue Summary:** On Wednesday 27 November 2024 a subset of Spatial Cloud environments experienced performance degradation for Map Manager, Tile Engine and Config Manager. Often presenting with errors at times that could be clicked through. **Root Cause Analysis:** At 08.50am the storage was increased on the secondary Storage system as a review indicated it was running out of space. This caused the throughput to sit at 100%, leading to performance and load issues. The throughput was high because the storage increase took place during Business-As-Usual \(BAU\) operations. **Corrective Measures:** Monitored the storage optimization until completion, which completed at 01:23 PM on 28 November, after which performance and loading returned to normal. **Preventive Measures:** Reviewed the standard operating procedure in line with alert refinement.

Read the full incident report →

Started Nov 04, 2024, 10:43 AM UTC· resolved Nov 04, 2024, 02:33 PM UTC

1. identified Nov 04, 2024, 10:43 AM UTC

   Our team has identified a fix which is being implemented now for Timetabling & Scheduling Non Production environments in the UK. The bulk of customer environments are resolved, with work ongoing to correct the remaining systems.

2. monitoring Nov 04, 2024, 11:31 AM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Nov 04, 2024, 11:31 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Nov 04, 2024, 02:33 PM UTC

   This incident has been resolved.

5. postmortem Nov 18, 2024, 11:27 PM UTC

   **Issue Summary:** On Monday 04 Nov 2024 at 9:50 it was noted that customers experienced performance degradation within non-production environments for T&S applications. The TechnologyOne team began an investigation immediately and quickly identified a fix. **Root Cause Analysis:** Servers found to be in an error state due to upstream maintenance which had an adverse affect on running services for the non-production environments. **Corrective Measures:** Manual intervention was required to failover services to return normal service. **Preventive Measures:** Enhancements have been identified to increase the resilience of running services. Additional proactive alerting has been created to attempt to predict the issues recurring in the future.

Read the full incident report →