TechnologyOne Outage History

Started Oct 01, 2026, 02:52 AM UTC· resolved Oct 01, 2026, 05:07 AM UTC

1. identified Oct 01, 2026, 02:52 AM UTC

   Our team has identified an issue affecting DP background jobs and slow performance for a subset of customers in ANZ Region. A fix is currently underway We anticipate the implementation of the fix to take 20 minutes to complete and verify. Users should see improvement within the next hour. We will provide a further update in 90 minutes or less.

2. monitoring Oct 01, 2026, 04:14 AM UTC

   Our team has verified the implementation of a fix is complete for the ANZ region and subset of customers. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

3. resolved Oct 01, 2026, 05:07 AM UTC

   After 1 hour monitoring, we have confirmed this incident is now resolved and services impacted are back to normal for the subset of impacted customers. We apologise for how you and your business may have been affected by this incident.

4. postmortem Oct 15, 2026, 04:00 AM UTC

   **Issue Summary:** At 12:55 PM on 01 October 2026, some customers reported experiencing significant delays in data processing \(DP\) jobs. This resulted in degraded system performance and a large backlog of submitted actions remaining in a 'pending' or 'submitted' state for several hours. **Root Cause Analysis:** The issue was caused by an overload of underlying platform components due to a sudden, very high volume of simultaneous requests, which temporarily exceeded available processing capacity. **Corrective Action:** Technology1 observed the delays occurring and the system automatically scaled up the background job processor's capacity to handle the increased load. While the auto-scaling mechanisms successfully provisioned the necessary resources, the sheer volume of the accumulated backlog required significant time to clear. The system continuously processed the jobs at this expanded capacity until all pending actions were resolved and the queue returned to normal levels. **Preventative Measures** * Optimize auto-scaling thresholds and improve monitoring to react even faster to unexpected spikes in demand. * Implement additional flow controls to manage high processing volumes and ensure system settings remain within safe operating limits.

Read the full incident report →

Started Sep 01, 2026, 02:27 AM UTC· resolved Sep 02, 2026, 04:02 AM UTC

1. investigating Sep 01, 2026, 02:27 AM UTC

   We are investigating an issue impacting service for a subset of ANZ customers. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. investigating Sep 01, 2026, 02:43 AM UTC

   A subset of customers across multiple releases in the ANZ region are experiencing Null Token errors and degraded access to CI and CI Anywhere. Our team is currently investigating and troubleshooting the issue. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

3. investigating Sep 01, 2026, 03:35 AM UTC

   We are continuing to investigate the issue affecting a subset of customers across multiple releases in the ANZ region. Impacted users may continue to intermittently experience Null Token errors when accessing CI. Access to CiA has improved. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

4. investigating Sep 01, 2026, 04:31 AM UTC

   We are continuing to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI across multiple releases in the ANZ region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

5. investigating Sep 01, 2026, 05:33 AM UTC

   We are still continuing to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI across multiple releases in the ANZ region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

6. investigating Sep 01, 2026, 06:36 AM UTC

   We are continuing to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI and Time tabling & Scheduling across multiple releases in the ANZ and UK region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

7. investigating Sep 01, 2026, 07:33 AM UTC

   We are working with the upstream provider to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI and Time tabling & Scheduling across multiple releases in the ANZ and UK region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

8. monitoring Sep 01, 2026, 08:25 AM UTC

   We have seen error rates reduce significantly and are continuing to monitor the platform closely. Current indicators show improved service stability, and our teams remain engaged to ensure performance continues to trend in the right direction. We will maintain enhanced monitoring through tomorrow and provide our next update by 9:00 AM AEST, or earlier if required.

9. monitoring Sep 01, 2026, 10:33 PM UTC

   We continue to observe very low error rates and stable performance. Monitoring remains in place, and our teams are continuing to closely observe the environment to ensure service stability is maintained. We will provide our next update by 2:00 PM AEST, or earlier if required.

10. resolved Sep 02, 2026, 04:02 AM UTC

    Following an extended period of stable performance and continued monitoring, this incident has now been resolved. Error rates have remained at expected levels, and no further customer impact has been observed. Our teams will continue to review the incident and implement any identified improvements as part of our standard post-incident process. We appreciate your patience and understanding while we worked to restore and validate service stability.

11. postmortem Sep 15, 2026, 01:15 AM UTC

    ### Incident Summary Between **31 August 2026 and 2 September 2026**, some customers experienced intermittent service disruptions, including slow application responsiveness, connection timeouts, and periods of Ci unavailability. The issue was caused by internal DNS resolution failures within an underlying CiA cluster, which affected communication between application services and backend caching components. Service stability has been restored and all affected environments are operating normally. ### Root Cause A significant increase in internal DNS requests related to Redis caching placed unexpected demand on the cluster's DNS service \(CoreDNS\). This resulted in memory exhaustion within CoreDNS, preventing it from scaling effectively to meet demand. As a result, some application services experienced delayed or failed DNS lookups, leading to connection timeouts and degraded application performance. ### Corrective Actions Taken To restore service and stabilise the platform, we: * Increased CoreDNS capacity by scaling the service. * Rebalanced workloads by moving affected environments to alternative CiA clusters. * Restarted impacted DNS services and closely monitored DNS performance until normal operation was confirmed. ### Preventative Actions * Optimising Redis connection management and DNS resolution behaviour to reduce DNS query volumes. * Reviewing and refining CoreDNS autoscaling policies, resource limits, and capacity thresholds. * Assessing long-term cluster capacity and workload distribution strategies. * Enhancing monitoring and alerting to identify and respond to DNS-related issues before customer impact occurs.

Read the full incident report →

Started Aug 31, 2026, 01:12 AM UTC· resolved Aug 31, 2026, 05:43 AM UTC

1. investigating Aug 31, 2026, 01:12 AM UTC

   We are investigating an issue impacting service for ANZ. A subset of customers in ANZ are experiencing intermittent issues. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. monitoring Aug 31, 2026, 01:21 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

3. monitoring Aug 31, 2026, 01:32 AM UTC

   We are continuing to monitor for any further issues.

4. monitoring Aug 31, 2026, 03:29 AM UTC

   We are continuing to monitor for another 2 hours to ensure no further customers are impacted.

5. resolved Aug 31, 2026, 05:43 AM UTC

   After considerable monitoring, this incident is now resolved. We will undertake a Post Incident Review and findings will be posted here on completion. We apologise for how you and your business may have been affected by this incident. Please contact the TechnologyOne Support team if you continue to experience issues.

6. postmortem Sep 15, 2026, 01:14 AM UTC

   ### Incident Summary Between **31 August 2026 and 2 September 2026**, some customers experienced intermittent service disruptions, including slow application responsiveness, connection timeouts, and periods of Ci unavailability. The issue was caused by internal DNS resolution failures within an underlying CiA cluster, which affected communication between application services and backend caching components. Service stability has been restored and all affected environments are operating normally. ### Root Cause A significant increase in internal DNS requests related to Redis caching placed unexpected demand on the cluster's DNS service \(CoreDNS\). This resulted in memory exhaustion within CoreDNS, preventing it from scaling effectively to meet demand. As a result, some application services experienced delayed or failed DNS lookups, leading to connection timeouts and degraded application performance. ### Corrective Actions Taken To restore service and stabilise the platform, we: * Increased CoreDNS capacity by scaling the service. * Rebalanced workloads by moving affected environments to alternative CiA clusters. * Restarted impacted DNS services and closely monitored DNS performance until normal operation was confirmed. ### Preventative Actions * Optimising Redis connection management and DNS resolution behaviour to reduce DNS query volumes. * Reviewing and refining CoreDNS autoscaling policies, resource limits, and capacity thresholds. * Assessing long-term cluster capacity and workload distribution strategies. * Enhancing monitoring and alerting to identify and respond to DNS-related issues before customer impact occurs.

Read the full incident report →

Started Jun 09, 2026, 03:15 PM UTC· resolved Jun 09, 2026, 08:28 PM UTC

1. investigating Jun 09, 2026, 03:15 PM UTC

   We are investigating an issue impacting service for ANZ Region. A subset of customers in ANZ Region are experiencing issues connecting to SFTP service. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Jun 09, 2026, 04:21 PM UTC

   Our team has identified a fix which is being implemented now for ANZ Region. We anticipate the implementation of the fix to take ~120 minutes to complete and verify.

3. identified Jun 09, 2026, 05:19 PM UTC

   Our team continue to deploy a fix which is being implemented now for ANZ Region, impacting SFTP. We anticipate the implementation of the fix to take 60 minutes to complete and verify.

4. monitoring Jun 09, 2026, 06:21 PM UTC

   Our team has verified the implementation of a fix is complete SFTP for ANZ Region. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

5. resolved Jun 09, 2026, 08:28 PM UTC

   After 2 hours monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

6. postmortem Jun 23, 2026, 12:27 AM UTC

   **Issue Summary** A subset of customers in the ANZ region experienced issues connecting to the SFTP service. **Root Cause** The service interruption was caused by an infrastructure deployment issue. During scheduled maintenance, an incomplete infrastructure configuration attempted to reconcile and rebuild components, resulting in the temporary unavailability of the SFTP service. **Corrective Action** To restore service, additional infrastructure capacity was provisioned in the ANZ region. A new SFTP host was deployed, the database was restored, and traffic routing was re-established to return customers to normal service. **Preventative Measures** As part of our ongoing resilience and reliability program, we have implemented additional monitoring and alerting, enhanced backup and recovery procedures, and are introducing Blue-Green deployment practices for the SFTP service to reduce the risk of future service disruptions.

Read the full incident report →

Started Feb 10, 2026, 09:41 PM UTC· resolved Feb 11, 2026, 04:35 AM UTC

1. identified Feb 10, 2026, 09:41 PM UTC

   We are investigating an issue impacting service for ANZ customers where environments are not available. The next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Feb 10, 2026, 09:59 PM UTC

   We have engaged with our upstream provider as this disruption is the result of an AWS outage. We are continuing to work with them to restore services. The next update will be provided in 60 minutes, or sooner if new information becomes available.

3. monitoring Feb 10, 2026, 10:12 PM UTC

   Our upstream provider has implemented a fix and noted that recovery is underway. We have noted that environments appear to be back online and operational. Please try accessing them again if you have not already. We will continue to monitor for the next 2 hours.

4. monitoring Feb 11, 2026, 12:17 AM UTC

   Our upstream provider has implemented a fix and confirmed that recovery is underway. We’re seeing environments come back online and operate normally. Full recovery efforts are still in progress to ensure there is no additional impact. Please try accessing the environments again if you haven’t already. We will continue to monitor for the next 2 hours.

5. monitoring Feb 11, 2026, 02:22 AM UTC

   Our upstream provider has advised the complete restoration globally continues. We have not observed any further issues however we will continue to monitor until the upstream provider confirms full restoration.

6. resolved Feb 11, 2026, 04:35 AM UTC

   The upstream provider has confirmed complete restoration and provided these details: "We experienced elevated DNS resolution errors for CloudFront distributions served from a subset of edge locations globally. During this time, customers may have received NXDOMAIN responses. Engineers were automatically engaged, immediately began investigating in multiple parallel paths and mitigated errors by taking the affected fleet of DNS servers out of service. We are confident this issue will not reoccur."

Read the full incident report →

Started Nov 08, 2025, 09:01 PM UTC· resolved Nov 09, 2025, 05:14 AM UTC

1. investigating Nov 08, 2025, 09:01 PM UTC

   The TechnologyOne Customer Community is currently unavailable. Should you need to contact support please phone on: Australia: 1300 735 130 New Zealand (free call): 0800 174 091 Malaysia (free call): 1800 818 451 United Kingdom: +44 (1223) 607860 Fiji +61 7 3167 7714 Papua New Guinea: +61 7 3167 7714 We will provide an update when the site is available again.

2. monitoring Nov 09, 2025, 03:09 AM UTC

   The Customer Community is available again. We will monitor for a few hours before setting this to resolved.

3. resolved Nov 09, 2025, 05:14 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 29, 2025, 07:26 PM UTC· resolved Oct 30, 2025, 12:50 AM UTC

1. monitoring Oct 29, 2025, 07:26 PM UTC

   We are aware of a service incident currently affecting Microsoft Azure, which may cause intermittent issues with authentication (SSO) if you are using this service to authenticate into your environment. While not all customers will be impacted, you may experience slower login times or temporary connection issues. We are actively monitoring the situation and will provide updates as more information becomes available. For real-time updates from Microsoft, please refer to the official Azure Status page: https://azure.status.microsoft/en-us/status

2. resolved Oct 30, 2025, 12:50 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 30, 2025, 06:36 AM UTC· resolved Sep 30, 2025, 09:05 AM UTC

1. investigating Sep 30, 2025, 06:36 AM UTC

   We are investigating an issue impacting service for ANZ Region / All Releases. A subset of customers in ANZ Region for All Releases are experiencing intermittent login issue Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Sep 30, 2025, 07:10 AM UTC

   Our team has identified a fix which is being implemented now for ANZ Region / All Releases. This impacted non-production environments predominantly, with various errors observed such as 503 and service unavailable messages.

3. monitoring Sep 30, 2025, 07:59 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region / All Releases. We will monitor the logs for the next 1 hour to ensure no further customers are impacted.

4. resolved Sep 30, 2025, 09:05 AM UTC

   After an hour monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

5. postmortem Oct 13, 2025, 05:58 AM UTC

   Issue Summary: On Tuesday, September 30, 2025, a subset of customers in the ANZ Region experienced sub-optimal performance and connection errors in Ci Anywhere / Ci. Root Cause: The degradation was caused by an upstream server going offline due to high resource usage. Corrective Action: Affected customers were migrated to alternative infrastructure to restore the service and the upstream provider fixed the impaired server Preventive Measure: As part of our ongoing resilience program, there is a focus to optimise resource efficiency.

Read the full incident report →

Started Sep 23, 2025, 07:48 PM UTC· resolved Sep 23, 2025, 11:12 PM UTC

1. investigating Sep 23, 2025, 07:48 PM UTC

   We are investigating an issue impacting service for our ANZ Region. A subset of customers in ANZ Region for are unable to access their environments. Customers may see a maintenance message when trying to access. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Sep 23, 2025, 08:03 PM UTC

   Our team has identified a fix which is being implemented now for ANZ Region. We anticipate the implementation of the fix to take 30 minutes to complete and verify.

3. monitoring Sep 23, 2025, 08:09 PM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

4. resolved Sep 23, 2025, 11:12 PM UTC

   This incident has been resolved.

5. postmortem Oct 08, 2025, 05:57 AM UTC

   Issue Summary: On Friday, September 24, 2025, a subset of customers in the ANZ Region experienced sub-optimal performance and connection errors in Ci Anywhere / Ci. Root Cause: The degradation was caused by an upstream server going offline due to high resource usage. Corrective Action: The upstream provider repaired the impaired servers to restore the service. Preventive Measure: As part of our ongoing resilience program, there is a focus to optimise resource efficiency.

Read the full incident report →

Started Aug 25, 2025, 02:04 AM UTC· resolved Aug 25, 2025, 03:35 AM UTC

1. investigating Aug 25, 2025, 02:04 AM UTC

   We are investigating an issue impacting service for ANZ Region / 2024A Release (Retired Release). Impact/Error/How to verify: A subset of customers in ANZ Region for 2024A Release are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. monitoring Aug 25, 2025, 02:23 AM UTC

   Areas noted to be impacted include Web Services, XLOne, Crystal Reports, ETL, Connected Content, Analytics, Leave, Organisation Management, and ECM. Our team has verified the implementation of a fix is complete for ANZ Region / 2024A Release (Retired Release). We will monitor the logs for the next 1 hour to ensure no further customers are impacted.

3. resolved Aug 25, 2025, 03:35 AM UTC

   After 1 hour monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

4. postmortem Sep 05, 2025, 05:46 AM UTC

   Issue Summary: On August 25 at 11:14 AM, the alerting system detected a spike in web service requests impacting customers on 2024A \(retired release\). This resulted in slowness and failures across multiple applications including Web Services, XLOne, Crystal Reports, ETL, Connected Content, Analytics, Leave, Organisation Management, and ECM. Root cause: Significant increase in system requests leading to failures accessing application infrastructure. Corrective Action: To stabilize the service, high-volume traffic was redirected to a dedicated target group, effectively isolating the load and restoring normal operations. Preventative measure: To enhance system reliability, safeguards such as rate limiting are included on the roadmap.

Read the full incident report →

Started Aug 01, 2025, 04:55 AM UTC· resolved Aug 01, 2025, 10:51 AM UTC

1. investigating Aug 01, 2025, 04:55 AM UTC

   We are investigating an issue impacting the DP Service. We have observed sub-optimal performance for the DP Service. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. investigating Aug 01, 2025, 05:58 AM UTC

   Our team is continuing to investigate. There are DP jobs processing however a large backlog has formed. Users may receive errors if using the run now action from the DP job. The next update will be provided in 60 minutes, or sooner if new information becomes available.

3. identified Aug 01, 2025, 07:00 AM UTC

   The underlying issue has been identified and is entering recovery. DP jobs noted as "priority" are the current focus. Background, scheduled, and engine jobs will take longer to clear. We will provide another update in 60 minutes or sooner if new information becomes available.

4. identified Aug 01, 2025, 07:56 AM UTC

   The priority DP jobs queue has cleared and is fully recovered. The service for United Kingdom and GovCloud is fully recovered. The background, scheduled, and engine jobs backlog is now reducing for the rest of ANZ. We will provide another update in 60 minutes or sooner if new information becomes available.

5. monitoring Aug 01, 2025, 08:54 AM UTC

   The backlog of DP jobs (Scheduled + Engine) has vastly improved for ANZ. UK and GovCloud continue to be fully resolved. We will monitor until the backlog is complete for the rest of ANZ.

6. resolved Aug 01, 2025, 10:51 AM UTC

   After 1 hour of monitoring, this incident is now resolved. The ANZ backlog has now been processed. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future, and post here on completion. We apologise for how you and your business may have been affected by this incident.

7. postmortem Aug 15, 2025, 07:02 AM UTC

   Issue Summary: On 01 Aug at 02:29 PM, the alerting system detected issues with the central DP management service. The TechnologyOne Team took corrective action to restore normal operation of the DP service. During the incident, users experienced delays in processing DP jobs and errors when attempting to submit them. Root Cause Analysis: Increased latency and high error rate of DP management service due to overloading of underlying platform components. Corrective Measures: Optimised scaling on the system to reduce the load on the DP processing to restore the service. Preventive Measures: As part of the ongoing project the first enhancement is complete and rollout commenced this week. Further enhancements to the resilience and availability of the DP service is currently in QA and undergoing testing.

Read the full incident report →

Started Jul 21, 2025, 05:27 AM UTC· resolved Jul 21, 2025, 07:57 AM UTC

1. investigating Jul 21, 2025, 05:27 AM UTC

   We are investigating an issue impacting service for a subset of customers’ non-production environments in the ANZ region, related to the 2025A release. Impact/Error/How to verify: Service Unavailable or 503 Errors on non-production environments for 2025A. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. monitoring Jul 21, 2025, 05:39 AM UTC

   Our team has verified the implementation of a fix is complete. We will monitor the logs for the next 1 hour to ensure no further customers are impacted.

3. identified Jul 21, 2025, 05:55 AM UTC

   We can see some errors still present. Our team has identified the fix was only partially applied and are continuing to implement this now. We anticipate the implementation of the fix to take 30 minutes to complete and verify.

4. identified Jul 21, 2025, 06:25 AM UTC

   Our team has confirmed that 2025A non-production environments that were affected are now operational. We have, since the start of the incident, identified that a subset of 2024B non-production environments are experiencing the same 503 and service offline errors. We are in the process of applying the mitigation to these impacted environments on 2024B and expect this to be completed in the next 30 minutes.

5. monitoring Jul 21, 2025, 06:53 AM UTC

   Our team has verified the implementation of a fix is now complete for 2024B affected environments. We will monitor the logs for the next 1 hour to ensure no further customers are impacted.

6. resolved Jul 21, 2025, 07:57 AM UTC

   After 1 hour monitoring this incident is now resolved. We apologise for how you and your business may have been affected by this incident.

7. postmortem Aug 07, 2025, 05:20 AM UTC

   Issue Summary: On July 21 at 03:18 PM, customers reported service unavailability and 503 errors on the non-production environments. Root Cause: The incident was caused by failure in upstream service. Corrective actions: Affected customers were migrated to alternative infrastructure to restore the service and the upstream provider fixed the impaired service. Preventative Measures: We are enhancing the platform architecture to strengthen resilience and ensure more reliable integration with upstream services, reducing the risk of disruption from system-level processes.

Read the full incident report →

Started Jul 15, 2025, 01:09 AM UTC· resolved Jul 15, 2025, 03:06 AM UTC

1. investigating Jul 15, 2025, 01:09 AM UTC

   We are investigating an issue impacting service for ANZ Region / 2024B Release. Impact/Error/How to verify: A subset of customers in ANZ Region for 2024B Releases are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. monitoring Jul 15, 2025, 01:52 AM UTC

   Our team has verified that errors have reduced, and navigation has improved across impacted areas such as Student Management, Timesheets, App Builder, Expense Management, Cash Receipting, Performance Planning and P&R for ANZ Region / 2024B Release. We will monitor the logs for the next 1 hour to ensure no further customers are impacted.

3. resolved Jul 15, 2025, 03:06 AM UTC

   After 1 hour monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

4. postmortem Jul 29, 2025, 05:46 AM UTC

   Issue Summary: On July 15 at 10:48 AM, customers reported slow response time and 504 gateway timeouts within Student Management, Timesheets, App Builder, Expense Management, Cash Receipting, Performance Planning and P&R in CiA. Root Cause: The issue was caused by a significant increase in system requests, combined with a database conflict that temporarily blocked key processes. Corrective actions: High-traffic requests were isolated to a separate processing path, and the affected servers were recycled to restore normal service. Preventative Measures: Software changes are being implemented to better manage peak in request volumes.

Read the full incident report →

Started Jun 18, 2025, 02:55 AM UTC· resolved Jun 30, 2025, 07:16 AM UTC

1. investigating Jun 18, 2025, 02:55 AM UTC

   We’re currently investigating ongoing 502 and Service Unavailable errors affecting some environments following the recent Mainline update (Snapshot 2025A (380)). These errors are currently observed in Financials and Supply Chain functions. They are linked to exceptions triggered by specific user actions, which cause the affected service to shut down. TechnologyOne is actively remediating as incidents occur. Mitigations have been implemented to reduce the impact while we identify the exception pattern and develop a permanent fix. We will provide an update as soon as more information is available.

2. identified Jun 18, 2025, 06:59 AM UTC

   We’ve identified the exception pattern causing intermittent 502 and Service Unavailable errors in Financials and Supply Chain functions. A permanent fix is now under development, and mitigations remain in place to reduce impact in the meantime. We’ll provide further updates as the fix progresses.

3. monitoring Jun 18, 2025, 11:51 PM UTC

   Mitigations have been applied and are currently being monitored to ensure stability and reduce further impact. A permanent fix remains in development. We’ll provide an update once the fix is available or if anything changes. Thank You for your patience.

4. monitoring Jun 19, 2025, 11:28 PM UTC

   The permanent fix for the intermittent 502 and Service Unavailable errors in Financials and Supply Chain functions will be released to Fastline this evening 2025A (394). The fix will be included in Mainline as part of the scheduled update next weekend. You can view the deployment schedule here - https://technologyonecorp365.sharepoint.com/:x:/s/SP_KT43N9/EeB5USzWQRRKhWmWvKnzamsBriu6Zd7TgMHnrwONTJQN9w?e=zVJ6FX We’ll continue to monitor and will provide confirmation once the Mainline update is complete.

5. monitoring Jun 25, 2025, 11:11 PM UTC

   The permanent fix for the intermittent 502 and Service Unavailable errors in Financials and Supply Chain functions will be deployed to Mainline as Snapshot 2025A (394). according to the planned schedule. Schedule can be found here: https://technologyonecorp365.sharepoint.com/:x:/s/SP_KT43N9/EeB5USzWQRRKhWmWvKnzamsBriu6Zd7TgMHnrwONTJQN9w?e=zVJ6FX We will continue to monitor through Monday, 30 June, to ensure stability. Further updates will be provided if required.

6. resolved Jun 30, 2025, 07:16 AM UTC

   The permanent fix for the intermittent 502 and Service Unavailable errors in Financials and Supply Chain functions was deployed to Mainline Snapshot 2025A (394). Following monitoring through to Monday, 30 June, no further issues have been observed. This incident is now considered resolved. Thank you for your patience while we worked to address the issue.

7. postmortem Jul 28, 2025, 01:21 AM UTC

   Issue Summary: On June 18 at 10:17 AM, customers reported intermittent 502 and service unavailable errors within Financials and Supply Chain functions in CiA. Root Cause: A recursive loop was found to occur during the initialization process under certain conditions. Corrective actions: High-traffic requests were isolated to a separate processing path, and the affected servers were recycled to restore normal service. Preventative Measures: Software changes implemented to remove the recursive looping.

Read the full incident report →

Started May 14, 2025, 09:05 PM UTC· resolved May 15, 2025, 04:10 AM UTC

1. investigating May 14, 2025, 09:05 PM UTC

   We are investigating an issue impacting the Spatial service for ANZ Region / All Releases. Impact/Error/How to verify: A subset of customers in ANZ Region for All Spatial Releases are experiencing the spatial services being offline for all modules. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified May 14, 2025, 10:06 PM UTC

   Our team has identified a fix which is being implemented now for the subset of impacted Spatial customers. Our investigation to date has noted that the nightly update has not been run so when the service comes back online (our focus) the data may be out of date. We anticipate the implementation of the fix to take 2 hours to complete and verify.

3. monitoring May 14, 2025, 11:55 PM UTC

   Our team has verified the implementation of a fix is complete for the affected spatial customers and services. The overnight processing for these customers has not completed. This means that data may be out of date. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

4. investigating May 15, 2025, 12:50 AM UTC

   Our monitoring has shown that login errors are present. Our team is troubleshooting an issue impacting Spatial Services for a subset of customers. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

5. identified May 15, 2025, 01:16 AM UTC

   Our team has identified a fix which is being implemented now for the affected customers and service. We anticipate the implementation of the fix to take 60 minutes to complete and verify.

6. monitoring May 15, 2025, 01:57 AM UTC

   Our team has verified the implementation of a fix is complete for the affected spatial customers and services. Reminder: The overnight processing for these customers has not completed. This means that data may be out of date. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

7. resolved May 15, 2025, 04:10 AM UTC

   After 2 hours monitoring this incident is now resolved. We will undertake a Post Incident Review and findings will be posted here on completion. We apologise for how you and your business may have been affected by this incident. Please contact the TechnologyOne Support team if you continue to experience issues.

8. postmortem May 29, 2025, 06:09 AM UTC

   **Issue Summary:** On 14 May at 11:46 PM, a subset of customers in ANZ experienced errors while attempting to access Spatial Cloud due to an infrastructure failure. The issue was promptly detected by the alerting system, and affected customers moved to alternative infrastructure. Service to Spatial Cloud was restored by 15 May 02:00 PM. **Root Cause Analysis:** Failure in upstream service. **Corrective Measures:** Affected customers were migrated to alternative infrastructure to restore the service. **Preventive Measures:** A new infrastructure layer has been deployed to support customers and prevent future similar failures.

Read the full incident report →

Started May 06, 2025, 11:48 PM UTC· resolved May 07, 2025, 07:38 AM UTC

1. investigating May 06, 2025, 11:48 PM UTC

   We are investigating an issue impacting DP service for ANZ Region for All Releases. Impact/Error/How to verify: A subset of customers in ANZ Region for all Releases are experiencing degradation of DP service Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified May 07, 2025, 12:52 AM UTC

   Our team has implemented the intended fix. We are working through a confirmation and verification to ensure the fix is complete. The next update will be provided in 60 minutes, or sooner if new information becomes available.

3. identified May 07, 2025, 01:49 AM UTC

   We are continuing to work through stabilising Priority and background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.

4. identified May 07, 2025, 02:52 AM UTC

   The implementation of remediation is in progress. The next update will be provided in 60 minutes, or sooner if new information becomes available.

5. identified May 07, 2025, 04:10 AM UTC

   The remediation has been successfully implemented, and the priority DP jobs are back to normal. The background DP jobs backlog is being cleared out in the coming hours. The next update will be provided in 60 minutes, or sooner if new information becomes available.

6. identified May 07, 2025, 04:49 AM UTC

   We are seeing improvements in the processing of background jobs and are continuing to work through further stabilisation. The next update will be provided in 60 minutes, or sooner if new information becomes available.

7. identified May 07, 2025, 05:52 AM UTC

   We are seeing improvements in the processing of background jobs and are still continuing to work through further stabilisation. The next update will be provided in 60 minutes, or sooner if new information becomes available.

8. monitoring May 07, 2025, 06:33 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ and UK Region for all Releases. We will monitor the logs for the next 1 hour to ensure no further customers are impacted.

9. resolved May 07, 2025, 07:38 AM UTC

   After 1 hour monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

10. postmortem May 20, 2025, 06:52 AM UTC

    Issue Summary: On 07 May at 08:34 AM, the alerting system detected issues with the central DP management service. The TechnologyOne Team took corrective action to restore normal operation of the DP service. During the incident, users experienced delays in processing DP jobs and errors when attempting to submit them. Root Cause Analysis: Increased latency and high error rate of DP management service due to overloading of underlying platform components. Corrective Measures: Optimised scaling on the system to reduce the load on the DP processing to restore the service. Preventive Measures: As part of the ongoing project the following was delivered as part of this incident response: * Changes implemented to the central DP management service to increase isolation and reduce load on underlying platform components. * Further enhancements to the resilience and availability of the DP service have been accelerated.

Read the full incident report →

Started Apr 24, 2025, 11:53 PM UTC· resolved Apr 25, 2025, 09:00 PM UTC

1. identified Apr 24, 2025, 11:53 PM UTC

   Our team has identified the DP Processing is degraded and are implementing a fix. We anticipate the implementation of the fix to take 60 minutes to complete and verify.

2. identified Apr 25, 2025, 12:59 AM UTC

   Our team has implemented the intended fix. We are working through a confirmation and verification to ensure the fix is complete. The next update will be provided in 60 minutes, or sooner if new information becomes available.

3. identified Apr 25, 2025, 01:51 AM UTC

   We have confirmed that priority DP jobs are now processing normally. We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.

4. identified Apr 25, 2025, 02:51 AM UTC

   We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.

5. identified Apr 25, 2025, 03:53 AM UTC

   We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.

6. identified Apr 25, 2025, 04:57 AM UTC

   We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.

7. identified Apr 25, 2025, 05:57 AM UTC

   We have observed less errors and higher success rate for submitted jobs. We are continuing to work through stabilising background DP job processing. The next update will be provided in 60 minutes, or sooner if new information becomes available.

8. identified Apr 25, 2025, 06:58 AM UTC

   We are seeing improvements in the processing of background jobs and are continuing to work through further stabilisation. The next update will be provided in 60 minutes, or sooner if new information becomes available.

9. monitoring Apr 25, 2025, 07:57 AM UTC

   Our team has verified the implementation of a fix is complete. We can see the background queue processing is improving and starting to stabilise. We will monitor the logs overnight to ensure no further customers are impacted.

10. resolved Apr 25, 2025, 09:00 PM UTC

    After overnight hours monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

11. postmortem May 09, 2025, 06:32 AM UTC

    Issue Summary: At 07:28 AM on 25 April, a subset of customers reported DP jobs queuing or unable to be submitted. Users also experienced long running DP times due to the DP jobs taking longer to be picked up and processed. Root Cause Analysis: The service responsible for managing queue processing degraded due to a scaling event triggered by long-running processes exceeding queue and processing limits. This led to an excessive load on the automation service platform which contributed to further load on the DP service. Corrective Measures: Modifications to the DP service scaling. Adjusted traffic distribution to ease the load profile. Rapid scaled downing of DP services to reduce the load. Preventive Measures: Observability enhancement to identify early warning patterns to allow rapid remediation . An ongoing project is being accelerated to further enhance the resilience and availability of the DP service planned for completion by August 2025.

Read the full incident report →

Started Apr 11, 2025, 01:36 AM UTC· resolved Apr 11, 2025, 02:55 AM UTC

1. investigating Apr 11, 2025, 01:36 AM UTC

   We are investigating an issue impacting service for ANZ Region for 2024B Release. Impact/Error/How to verify: A subset of customers in ANZ Region for 2024B Release are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. monitoring Apr 11, 2025, 01:52 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region for 2024B Release. We will monitor the logs for the next 1 hours to ensure no further customers are impacted.

3. resolved Apr 11, 2025, 02:55 AM UTC

   After 1 hour monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

4. postmortem Apr 30, 2025, 04:39 AM UTC

   Issue Summary: At 09:01 AM on 11 April, a Subset of customer’s production datasets experienced performance-based errors. Root Cause: Due to a scheduling error, customers were incorrectly assigned to the latest release version due to incorrect filters selected during upgrade process. Corrective Actions: Customer’s datasets were rolled back to desired release version. Preventive Measures: The upgrade processes have been reviewed and enhanced to incorporate additional validations to ensure right filters are applied.

Read the full incident report →

Started Apr 03, 2025, 11:51 PM UTC· resolved Apr 04, 2025, 07:43 AM UTC

1. investigating Apr 03, 2025, 11:51 PM UTC

   We are investigating an issue impacting service for ANZ Region / 2025A. Impact/Error/How to verify: A subset of customers in ANZ Region for 25A Releases are experiencing 500 and 502 errors intermittently in CiAnywhere. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

2. monitoring Apr 04, 2025, 12:22 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region / 25A Releases. Please refresh/clear your browser cache or try a private/incognito browser if you still receive an error. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

3. investigating Apr 04, 2025, 01:08 AM UTC

   We have seen new errors arise since fix applied. We are investigating this further. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

4. investigating Apr 04, 2025, 01:37 AM UTC

   Our team is continuing to investigate. Impact/Error/How to verify: A subset of customers in ANZ Region for 25A Releases are experiencing 502 errors or Service Offline error intermittently in CiAnywhere. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

5. investigating Apr 04, 2025, 02:07 AM UTC

   Our team is continuing to investigate the issue, and our investigation is narrowing down. Impact/Error/How to verify: A subset of customers in ANZ Region for 2025A Releases are experiencing 502 errors or Service Offline error intermittently in CiAnywhere. Our investigation shows this is occurring for Financials and Supply Chain processors. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

6. identified Apr 04, 2025, 02:35 AM UTC

   Our team has identified the issue and have applied mitigations to reduce the errors received. We continue to work on a complete fix. The next update will be provided in 60 minutes, or sooner if new information becomes available.

7. identified Apr 04, 2025, 03:35 AM UTC

   Our logs show that errors do continue however at a reduced rate. We continue to work on a complete fix. The next update will be provided in 60 minutes, or sooner if new information becomes available.

8. identified Apr 04, 2025, 04:34 AM UTC

   Our logs show that errors are minimal right now. We continue to work on a fix for the underlying issue. The next update will be provided in 60 minutes, or sooner if new information becomes available.

9. identified Apr 04, 2025, 05:36 AM UTC

   Our logs show that errors are very limited now. We continue to work on a fix for the underlying issue. The next update will be provided in 60 minutes, or sooner if new information becomes available.

10. monitoring Apr 04, 2025, 05:43 AM UTC

    Our team has verified the implementation of a fix is complete. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

11. resolved Apr 04, 2025, 07:43 AM UTC

    After 2 hours monitoring, this incident is now resolved. We will perform a post incident review and post here on completion. We apologise for how you and your business may have been affected by this incident.

12. postmortem Apr 22, 2025, 06:54 AM UTC

    **Issue Summary** On 4 April 2025 a subset of customers experienced intermittent 502 and server unavailable errors from 8.32am AEST through to 1.35pm AEST. Initial analysis pointed to unhealthy servers causing the 502 and server unavailable errors. Impacted customer datasets were moved to a healthy server stack. The errors began to appear on the new server stack. Further analysis found a software issue causing a stack overflow when a user on a specific dataset performed a specific action. This dataset was isolated to ensure no further disruption to other customer datasets. **Root Cause** Stack overflow exception triggered by a specific workflow activity within the Data Entry app. **Corrective Actions** Replaced unhealthy servers with new servers and moved customer datasets to the healthy servers. Isolated customer dataset triggering the stack overflow exception. **Preventative Actions** Observability enhanced to identify any stack overflow exceptions and triggering application including an alert. Resolve underlying software issue to address the stack overflow exception.

Read the full incident report →

Started Feb 27, 2025, 06:20 AM UTC· resolved Feb 27, 2025, 12:10 PM UTC

1. investigating Feb 27, 2025, 06:20 AM UTC

   We are investigating an issue impacting the DP service for UK Region. This will present as: - Users see DP jobs as "submitted" - Users in a worksheet waiting on a DP job to be completed will see that part of the process is continuing to spin. We are applying mitigations and next update will be provided in 60 minutes, or sooner if new information becomes available.

2. investigating Feb 27, 2025, 07:17 AM UTC

   Our investigation continues for the UK DP services however the logs are showing promising progress. We will continue to apply mitigations, and next update will be provided in 60 minutes, or sooner if new information becomes available.

3. investigating Feb 27, 2025, 08:17 AM UTC

   Our investigation into the UK DP services is ongoing. We are currently examining logs for any errors and reviewing reports from customers. Next update will be provided in 60 minutes, or sooner if new information becomes available.

4. investigating Feb 27, 2025, 09:11 AM UTC

   Our logs show a large increase in new DP jobs progressing. We continue to apply mitigations, and the next update will be provided within 60 minutes or sooner.

5. monitoring Feb 27, 2025, 09:35 AM UTC

   Our team has verified the implementation of a fix is complete. We will monitor the logs for the next 2 hours to ensure no further impacts.

6. resolved Feb 27, 2025, 12:10 PM UTC

   After 2 hours monitoring, this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future, and post here on completion. We apologise for how you and your business may have been affected by this incident.

7. postmortem Mar 04, 2025, 08:11 AM UTC

   **Issue Summary:** On Thursday 27 February at 12.00am GMT alert monitoring indicated that our cloud orchestration platform was spiking above its normal response time. The TechnologyOne team began an investigation immediately. The impact was seen by users on DP jobs queuing or unable to be submitted. Users also experienced longer run times on worksheet processes due to the DP jobs taking longer to be picked up and processed. **Root Cause Analysis:** Queue and processing limits reached due to long-running processes locking the cloud orchestration database. This caused the CPU utilisation to max out at 100%. Whilst the cloud orchestration database was recovered within 45 mins the Cloud DP Service did not recover due to the backlog of DP jobs. The TechnologyOne team undertook several actions to clear the backlog from the Cloud DP Service and this was stabilised at 9.33am GMT. **Corrective Measures:** Restarted all the tasks supporting the Cloud DP. Built additional microservice clusters and scaled out the microservice cluster to handle the load. Recycled servers in the microservice cluster. Scaled back the number of DP Servers \(due to auto scaling\) to reduce the load. **Preventive Measures:** A full review of the Cloud DP service in conjunction with an upstream provider is underway with the expectation additional mitigations will be implemented. An ongoing project is being accelerated to further enhance the scalability and performance underload for the DP microservice and is planned for completion by August 2025.

Read the full incident report →

Started Feb 27, 2025, 03:01 AM UTC· resolved Feb 27, 2025, 12:09 PM UTC

1. investigating Feb 27, 2025, 03:01 AM UTC

   We are investigating an issue impacting the DP service for ANZ Region / All Releases. Impact/Error/How to verify: A subset of customers are experiencing DP jobs stalling or major delays. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. investigating Feb 27, 2025, 03:43 AM UTC

   Our team is continuing to investigate an issue impacting the DP service and identify a solution to resolve. We can see the impact for customers as follows: - Users see DP jobs as "submitted" - Users in a worksheet waiting on a DP job to be completed will see that part of the process is continuing to spin. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

3. investigating Feb 27, 2025, 04:42 AM UTC

   We have identified the root cause of the issue and have undertaken steps to mitigation. Our logs show that error rates on DP servers have decreased however we still see a large number of DP logs in a submitted state. This will be presenting as: - Users see DP jobs as "submitted" - Users in a worksheet waiting on a DP job to be completed will see that part of the process is continuing to spin. The next update will be provided within 60 minutes or sooner.

4. investigating Feb 27, 2025, 05:37 AM UTC

   Our logs continue to show that error rates on DP servers have decreased we can see DP jobs progressing however we also see growth in Submitted jobs. This will be presenting as: - Users see DP jobs as "submitted" - Users in a worksheet waiting on a DP job to be completed will see that part of the process is continuing to spin. We continue to apply mitigations, and the next update will be provided within 60 minutes or sooner.

5. investigating Feb 27, 2025, 06:33 AM UTC

   Our logs continue to show that error rates on DP servers have decreased we can see new DP jobs progressing however the queue for submitted jobs is growing. This issue will be presenting as: - Users see DP jobs as "submitted" - Users in a worksheet waiting on a DP job to be completed will see that part of the process is continuing to spin. - An error appears if user forces a job to run. We continue to apply mitigations, and the next update will be provided within 60 minutes or sooner.

6. investigating Feb 27, 2025, 07:31 AM UTC

   Our logs continue to show that error rates on DP servers have decreased we can see new DP jobs progressing over 46000 completed in the last hour however this is a much lower rate than normal. You may see this presenting as: - Users see DP jobs as "submitted" for longer time then normal. - Users in a worksheet waiting on a DP job to be completed will see that part of the process is continuing to spin. - An error appears if user forces a job to run. We continue to apply mitigations, and the next update will be provided within 60 minutes or sooner.

7. investigating Feb 27, 2025, 08:34 AM UTC

   Our logs show a large increase in new DP jobs progressing with over 90000 completed in the last hour. We continue to apply mitigations, and the next update will be provided within 60 minutes or sooner.

8. monitoring Feb 27, 2025, 09:33 AM UTC

   Our team has verified the implementation of a fix is complete. We will monitor the logs for the next 2 hours to ensure no further impacts.

9. resolved Feb 27, 2025, 12:09 PM UTC

   After 2 hours monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future, and post here on completion. We apologise for how you and your business may have been affected by this incident.

10. postmortem Mar 04, 2025, 08:08 AM UTC

    **Issue Summary:** On Thursday 27 February at 10.40am AEST alert monitoring indicated that our cloud orchestration platform was spiking above its normal response time. The TechnologyOne team began an investigation immediately. The impact was seen by users on DP jobs queuing or unable to be submitted. Users also experienced longer run times on worksheet processes due to the DP jobs taking longer to be picked up and processed. **Root Cause Analysis:** Queue and processing limits reached due to long-running processes locking the cloud orchestration database. This caused the CPU utilisation to max out at 100%. Whilst the cloud orchestration database was recovered within 45 mins the Cloud DP Service did not recover due to the backlog of DP jobs. The TechnologyOne team undertook several actions to clear the backlog from the Cloud DP Service and this was stabilised at 7.33pm AEST. **Corrective Measures:** Restarted all the tasks supporting the Cloud DP. Built additional microservice clusters and scaled out the microservice cluster to handle the load. Recycled servers in the microservice cluster. Scaled back the number of DP Servers \(due to auto scaling\) to reduce the load. **Preventive Measures:** A full review of the Cloud DP service in conjunction with an upstream provider is underway with the expectation additional mitigations will be implemented. An ongoing project is being accelerated to further enhance the scalability and performance underload for the DP microservice and is planned for completion by August 2025.

Read the full incident report →

Started Feb 17, 2025, 10:53 AM UTC· resolved Feb 17, 2025, 10:43 PM UTC

1. identified Feb 17, 2025, 10:53 AM UTC

   Our team has identified a subsequent issue to the outage earlier today for a subset of customers impacting Spatial Services We are currently applying fixes to the impacted environments with the goal to have the fix applied to all affected customers prior to the overnight processing commencing at 11.30PM. We anticipate the implementation of the fix to take 2 hours to complete and verify. The next update will be posted when the implementation of the fixes has been completed.

2. monitoring Feb 17, 2025, 12:57 PM UTC

   Our team has verified the implementation of a fix is complete for the impacted environments We will monitor the logs for the next few hours to ensure no further customers are impacted.

3. resolved Feb 17, 2025, 10:43 PM UTC

   After overnight monitoring this incident is now resolved. We will perform a post incident review to identify underlying cause, and preventive action to avoid a repeat in the future and post here on completion. We apologise for how you and your business may have been affected by this incident.

4. postmortem Feb 20, 2025, 06:47 AM UTC

   **Issue Summary** TechnologyOne observed system instability over the weekend in the spatial data layer after the regular storage maintenance by the upstream vendor and took measures to recover via a database failover on 16 February 2025. On 17 February 2025 a subset of spatial customers in the ANZ region were unable to access ConfigManager, Intramaps and Nearmaps. A portion of this customer base were restored by 2pm AEST with the remaining being brought online by 4pm. At 7pm AEST errors presented in the logs which identified further error states presenting. Action was taken to restore the affected databases from backup and all were restored by 11pm AEST. At 11.15pm AEST errors in replication were observed and found to be as a result of replication commencing at time restoration was underway. Access was restored and replications re-run with monitoring into the morning. **Root Cause** The database cluster became instable due to Continuous Availability \(CA\) configuration not being enabled at the spatial storage layer. This configuration prevents inconsistences for the database during storage maintenance. **Corrective Actions** A failover was conducted on the Sunday evening however many databases continued to enter recovery state. Corrected a storage permissions issue. Brought databases online one by one as each time attempted as a group failed. Restored subset of databases found to be in a corrupted state from back-up. **Preventative Actions** Enable CA on the database share on the primary storage for spatial. A maintenance window is planned between 8pm-9pm AEST on 22/2/2025. Updates will be provided via the status page.

Read the full incident report →

Started Feb 16, 2025, 08:55 PM UTC· resolved Feb 17, 2025, 08:33 AM UTC

1. investigating Feb 16, 2025, 08:55 PM UTC

   Our team is troubleshooting an issue impacting Spatial Services for ANZ Region / ALL Releases. Errors are presenting for IntraMaps and Config Manager. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Feb 16, 2025, 09:50 PM UTC

   Our team has identified a fix which is being implemented now. We anticipate the implementation of the fix for all impacted customers to take up to 4 hours to complete and verify.

3. identified Feb 16, 2025, 11:20 PM UTC

   Our team has restored Spatial Services to a number of impacted customers across the ANZ region. We are continuing to apply the fix to the remaining customers. We will provide another update in 90 minutes, or sooner if new information becomes available.

4. identified Feb 17, 2025, 12:59 AM UTC

   We are continuing to apply a fix to the remaining impacted customers. We will provide another update in 60 minutes, or sooner if new information becomes available.

5. identified Feb 17, 2025, 02:00 AM UTC

   We are continuing to apply a fix to the remaining impacted customers. As our incident has progressed we have found 2 separate issues. The first issue has been resolved and impacted customers restored. For the remaining customers impacted this relates to a 2nd issue which we are continuing to work through applying a fix for and we anticipate this will take 4 hours to restore the remaining customers.

6. identified Feb 17, 2025, 04:02 AM UTC

   We are continuing to apply a fix to the remaining impacted customers. We will provide another update in 60 minutes, or sooner if new information becomes available.

7. identified Feb 17, 2025, 04:56 AM UTC

   Our team has restored Spatial Services to most of the impacted customers across the ANZ region. We are continuing to apply a fix to the remaining customers. We will provide another update in 60 minutes, or sooner if new information becomes available.

8. monitoring Feb 17, 2025, 06:25 AM UTC

   Our team has verified the implementation of a fix is completed for all impacted customers now. As we have brought each of the Spatial Services back online, we have noted some performance degradation for customers due to timeouts for those not yet online. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

9. resolved Feb 17, 2025, 08:33 AM UTC

   After 2 hours monitoring this incident is now resolved. We will undertake a Post Incident Review and findings will be posted here on completion. We apologise for how you and your business may have been affected by this incident.

10. postmortem Feb 20, 2025, 06:47 AM UTC

    **Issue Summary** TechnologyOne observed system instability over the weekend in the spatial data layer after the regular storage maintenance by the upstream vendor and took measures to recover via a database failover on 16 February 2025. On 17 February 2025 a subset of spatial customers in the ANZ region were unable to access ConfigManager, Intramaps and Nearmaps. A portion of this customer base were restored by 2pm AEST with the remaining being brought online by 4pm. At 7pm AEST errors presented in the logs which identified further error states presenting. Action was taken to restore the affected databases from backup and all were restored by 11pm AEST. At 11.15pm AEST errors in replication were observed and found to be as a result of replication commencing at time restoration was underway. Access was restored and replications re-run with monitoring into the morning. **Root Cause** The database cluster became instable due to Continuous Availability \(CA\) configuration not being enabled at the spatial storage layer. This configuration prevents inconsistences for the database during storage maintenance. **Corrective Actions** A failover was conducted on the Sunday evening however many databases continued to enter recovery state. Corrected a storage permissions issue. Brought databases online one by one as each time attempted as a group failed. Restored subset of databases found to be in a corrupted state from back-up. **Preventative Actions** Enable CA on the database share on the primary storage for spatial. A maintenance window is planned between 8pm-9pm AEST on 22/2/2025. Updates will be provided via the status page.

Read the full incident report →

Started Feb 10, 2025, 03:38 AM UTC· resolved Feb 10, 2025, 06:29 AM UTC

1. monitoring Feb 10, 2025, 03:38 AM UTC

   Our team has verified this disruption is resolved for the subset of customers in ANZ Region for all Releases. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

2. monitoring Feb 10, 2025, 03:39 AM UTC

   We are continuing to monitor for any further issues.

3. investigating Feb 10, 2025, 03:50 AM UTC

   We are investigating as our monitoring identified reoccurrence of the issue impacting service for a subset of customer in ANZ Region for all Releases. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

4. identified Feb 10, 2025, 03:55 AM UTC

   Our team has identified a fix which is being implemented now. We anticipate the implementation of the fix to take 30 minutes to complete and verify.

5. monitoring Feb 10, 2025, 04:21 AM UTC

   Our team has verified the implementation of the fix is complete. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

6. resolved Feb 10, 2025, 06:29 AM UTC

   After 2 hours of monitoring this incident is now resolved. We apologise for how you and your business may have been affected by this incident.

7. postmortem Feb 13, 2025, 11:12 PM UTC

   **Issue Summary** On 10 February 2025 a partial failure in the network occurred at 1.00pm AEST. This impacted a subset of customers, like those impacted on 3 February 2025, who then experienced a range of 5XX errors, and many users were unable to login. This disruption lasted for 2-3 mins for Production and 3-5 mins for non-production and auto recovered and then reoccurred 30 minutes later with a further 2-3 min disruption for Production and 3-5 min disruption for non-production and again auto recovered. Preventative actions began being applied progressively across regions from 2pm on 10/2/2025. **Root Cause** A partial failure on managed DNS caused network instability leading to intermittent access to application infrastructure. **Corrective Actions** The managed DNS was automatically recovered. **Preventative Actions** Implemented configuration changes in managed DNS based on recommendations from our upstream provider who reviewed the workload patterns to provide additional resilience to the service.

Read the full incident report →

Started Feb 04, 2025, 01:23 AM UTC· resolved Feb 04, 2025, 05:33 AM UTC

1. investigating Feb 04, 2025, 01:23 AM UTC

   We are investigating an issue impacting service for ANZ Region for all Releases. A subset of customers in ANZ Region for all Release are experiencing 504 gateway errors. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. identified Feb 04, 2025, 01:35 AM UTC

   Our team has identified a fix which is being implemented now. We anticipate the implementation of the fix to take 60 minutes to complete and verify.

3. monitoring Feb 04, 2025, 02:37 AM UTC

   Our team has verified the implementation of a fix is complete for the impacted production environments. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

4. resolved Feb 04, 2025, 05:33 AM UTC

   After a few hours monitoring this incident is now resolved. We apologise for how you and your business may have been affected by this incident.

5. postmortem Feb 13, 2025, 11:11 PM UTC

   **Issue Summary** On 4 February 2025 a partial failure in the network occurred at 11.00am AEST. This impacted a subset of customers, different to those impacted on 3 February 2025, who then experienced a range of 5XX errors intermittently. This disruption was up to 60 minutes. TechnologyOne team immediately applied mitigations developed on 3 February 2025 to recover. TechnologyOne teams continued to work and analyse the root cause and develop actions through to 10/2/2025 Preventative actions began being applied progressively across regions from 2pm on 10/2/2025. **Root Cause** A partial failure on managed DNS caused network instability leading to intermittent access to application infrastructure. **Corrective Actions** Undertook the mitigations \(developed on 3rd February\) to recover. The managed DNS was recovered by our upstream vendor. **Preventative Actions** Implemented configuration changes in managed DNS based on recommendations from our upstream provider who reviewed the workload patterns to provide additional resilience to the service.

Read the full incident report →