TechnologyOne incident
DP Service Disruption for a subset of customers in ANZ and UK Region for all Releases
TechnologyOne experienced a major incident on May 6, 2025 affecting Database Services and Database Services, lasting 7h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 06, 2025, 11:48 PM UTC
We are investigating an issue impacting DP service for ANZ Region for All Releases. Impact/Error/How to verify: A subset of customers in ANZ Region for all Releases are experiencing degradation of DP service Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified May 07, 2025, 12:52 AM UTC
Our team has implemented the intended fix. We are working through a confirmation and verification to ensure the fix is complete. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified May 07, 2025, 01:49 AM UTC
We are continuing to work through stabilising Priority and background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified May 07, 2025, 02:52 AM UTC
The implementation of remediation is in progress. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified May 07, 2025, 04:10 AM UTC
The remediation has been successfully implemented, and the priority DP jobs are back to normal. The background DP jobs backlog is being cleared out in the coming hours. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified May 07, 2025, 04:49 AM UTC
We are seeing improvements in the processing of background jobs and are continuing to work through further stabilisation. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified May 07, 2025, 05:52 AM UTC
We are seeing improvements in the processing of background jobs and are still continuing to work through further stabilisation. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- monitoring May 07, 2025, 06:33 AM UTC
Our team has verified the implementation of a fix is complete for ANZ and UK Region for all Releases. We will monitor the logs for the next 1 hour to ensure no further customers are impacted.
- resolved May 07, 2025, 07:38 AM UTC
After 1 hour monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.
- postmortem May 20, 2025, 06:52 AM UTC
Issue Summary: On 07 May at 08:34 AM, the alerting system detected issues with the central DP management service. The TechnologyOne Team took corrective action to restore normal operation of the DP service. During the incident, users experienced delays in processing DP jobs and errors when attempting to submit them. Root Cause Analysis: Increased latency and high error rate of DP management service due to overloading of underlying platform components. Corrective Measures: Optimised scaling on the system to reduce the load on the DP processing to restore the service. Preventive Measures: As part of the ongoing project the following was delivered as part of this incident response: * Changes implemented to the central DP management service to increase isolation and reduce load on underlying platform components. * Further enhancements to the resilience and availability of the DP service have been accelerated.