TechnologyOne experienced a minor incident on April 24, 2025 affecting Batch Services (DP Jobs) and Batch Services (DP Jobs) and 1 more component, lasting 21h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 24, 2025, 11:53 PM UTC
Our team has identified the DP Processing is degraded and are implementing a fix. We anticipate the implementation of the fix to take 60 minutes to complete and verify.
- identified Apr 25, 2025, 12:59 AM UTC
Our team has implemented the intended fix. We are working through a confirmation and verification to ensure the fix is complete. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified Apr 25, 2025, 01:51 AM UTC
We have confirmed that priority DP jobs are now processing normally. We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified Apr 25, 2025, 02:51 AM UTC
We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified Apr 25, 2025, 03:53 AM UTC
We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified Apr 25, 2025, 04:57 AM UTC
We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified Apr 25, 2025, 05:57 AM UTC
We have observed less errors and higher success rate for submitted jobs. We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified Apr 25, 2025, 06:58 AM UTC
We are seeing improvements in the processing of background jobs and are continuing to work through further stabilisation. The next update will be provided in 60 minutes, or sooner if new information becomes available.
- monitoring Apr 25, 2025, 07:57 AM UTC
Our team has verified the implementation of a fix is complete. We can see the background queue processing is improving and starting to stabilise. We will monitor the logs overnight to ensure no further customers are impacted.
- resolved Apr 25, 2025, 09:00 PM UTC
After overnight hours monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.
- postmortem May 09, 2025, 06:32 AM UTC
Issue Summary: At 07:28 AM on 25 April, a subset of customers reported DP jobs queuing or unable to be submitted. Users also experienced long running DP times due to the DP jobs taking longer to be picked up and processed. Root Cause Analysis: The service responsible for managing queue processing degraded due to a scaling event triggered by long-running processes exceeding queue and processing limits. This led to an excessive load on the automation service platform which contributed to further load on the DP service. Corrective Measures: Modifications to the DP service scaling. Adjusted traffic distribution to ease the load profile. Rapid scaled downing of DP services to reduce the load. Preventive Measures: Observability enhancement to identify early warning patterns to allow rapid remediation . An ongoing project is being accelerated to further enhance the resilience and availability of the DP service planned for completion by August 2025.