TechnologyOne Outage History

**Issue Summary:** On Monday 04 Nov 2024 at 9:50 it was noted that customers experienced performance degradation within non-production environments for T&S applications. The TechnologyOne team began an investigation immediately and quickly identified a fix. **Root Cause Analysis:** Servers found to be in an error state due to upstream maintenance which had an adverse affect on running services for the non-production environments. **Corrective Measures:** Manual intervention was required to failover services to return normal service. **Preventive Measures:** Enhancements have been identified to increase the resilience of running services. Additional proactive alerting has been created to attempt to predict the issues recurring in the future.

**Issue Summary:** On Monday 21 October 2024 at 11.15am alert monitoring indicated that our cloud orchestration platform was spiking above it's normal response time. The TechnologyOne team began an investigation immediately. The impact was most noticeable with DP jobs queuing or unable to be submitted **Root Cause Analysis:** Queue and processing limits reached due to long-running processes locking the database. This caused the CPU utilisation to max out at 100%. **Corrective Measures:** Restarted all the tasks supporting the Cloud DP. Scaled out the microservice cluster to handle the load. **Preventive Measures:** Review and refine existing monitoring alerts. Longer term project to further enhance the scalability and performance underload for the DP microservice.

**Issue Summary** On 30/09/2024, a subset of users experienced errors when attempting to access Timetabling and Scheduling. This typically manifested a browser specific error message. **Root Cause Analysis** The incident was triggered by a change to the network access control list as part of a planned and pre-tested change. **Corrective Actions** Reverted to the prior configuration. **Preventive Measures** Built a programmatic check for configuration update to templates which will be run on any future updates.

**Issue Summary** On 5/09/2024, a subset of users experienced 404 errors while attempting to Access Timetabling and Scheduling. **Root Cause Analysis** The incident was triggered by Windows Defender Antivirus erroneously flagging a part of the Timetabling software as potentially harmful. TechnologyOne software is checked rigorously and on review, no threat or issue existed inside the software. **Corrective Actions** 1. Cleared the warning flag on affected servers returning them to normal operation 2. Flagged software added to a scan exclusion list in future builds. **Preventive Measures** Exclude pre-checked and known-safe software from more aggressive anti-virus protection mechanisms.

**Issue Summary** On 25/07/2024, a subset of users were unable to successfully connect to Ci services and was presented with a 404 or 504 Errors **Root Cause Analysis** The incident was triggered by a third-party software update release that led to affected servers rejecting new login requests. This issue had a flow-on effect on the remaining stable servers increased beyond their capacity, leading to additional service disruptions. #### **Corrective Actions** 1. The servers that were rejecting login requests were replaced to restore service functionality. 2. A dedicated patch was developed and deployed across all impacted servers to address the underlying issue caused by the third-party software update. #### **Preventive Measures** Workshops for third-party update deployments are currently being conducted to develop a strategy that ensures customer stabilization

## Issue Summary On 17/06/2024, multiple customers were presented with Error 502 and 504 when their users attempted to load into CI in the cloud \(Production\). ## Root Cause Analysis A configuration error in scaling values during routine maintenance. ## Corrective Actions Scaling values were increased to accommodate customers' size and scale, with servers updated to adopt these values. ## Preventive Measures Standard operating procedures have been updated to ensure correct configuration is set for scaling moving forward.

**Issue Summary:** On 11/07/2024, a subset of customers based in the UK on versions 2023B and 2024A experienced a service degradation with File Access. Impacted customers were presented with an "Unexpected Error" message when accessing attachments in ECM. **Root Cause:** A configuration bug in ECM caused it to connect to an incorrect region. **Corrective Actions:** A rollback of the platform changes was executed to restore services and resume normal operations. **Preventive Measures:** A software update addressing this issue has been implemented and will be rolled out in the upcoming release \(2023B - Snapshot 675, 2024A - Snapshot 423\).

**Issue Summary** The Mainline upgrade for 2023B \(635\) and 2024A \(384\) was scheduled to be deployed to all mainline customers on the 29th of June 2024. For a subset of customer datasets, the upgrade failed to deploy, resulting in the need to reschedule these for upgrade the following day \(30th June 2024\). **Root Cause** * Initially a service disruption caused the upgrade process to fail to start for some customer data sets and complete for other customer data sets. * A subsequent degradation was found in the service causing slow response times which lead to incomplete upgrades. ‌ **Corrective Actions** 1. We rescheduled the upgrades for the affected customer datasets as necessary. ‌ **Preventive Actions** 1. Research options to automate a retry for the deployment service and implement. 2. Improve the queries that caused slow response times.

**Issue Summary:** On the 4th of June 2024, the SaaS operations team identified an increase in user traffic, which caused degraded performance primarily in CES, ECM and P&R. **Root Cause Analysis:** Queue and processing limits were reached due to an unexpected increase in traffic and resource demand. The issue was isolated to 2 customers. **Preventive Measures:** Previously implemented preventive measures worked as the impact was contained.