Service Degradation for a subset of customers in ANZ Region for all Release

Update timeline

1. investigating Feb 03, 2025, 12:38 AM UTC

   We are investigating an issue impacting service for ANZ Region for all Release. A subset of customers in ANZ Region for all Release are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. investigating Feb 03, 2025, 01:39 AM UTC

   Our team is continuing to investigate an issue impacting service for ANZ Region for All Releases. A subset of customers in ANZ Region for All Releases are experiencing sub-optimal performance / connection errors in Ci Anywhere / Ci. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

3. monitoring Feb 03, 2025, 02:02 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region and all Releases. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

4. monitoring Feb 03, 2025, 03:08 AM UTC

   Monitoring Update: We have observed the majority of Production environments have been restored and working again. For non-Production environments we continue to apply the fix. Next update expected in 60 mins.

5. monitoring Feb 03, 2025, 04:18 AM UTC

   The Production environments have been restored and working again. For non-Production environments we continue to apply the fix.

6. monitoring Feb 03, 2025, 05:23 AM UTC

   The Production environments have been restored and working again. The issue is intermittent in non-production environment, and we are working to stabilize Next update expected in 2 hours.

7. monitoring Feb 03, 2025, 07:10 AM UTC

   We are continuing to work on stabilising the non-production environments. Monitoring indicates the issues in non-production environments are intermittent. Monitoring confirms all Production environments have been restored and remain stable. Next update expected in 2 hours.

8. monitoring Feb 03, 2025, 09:06 AM UTC

   A fix has been implemented for the impacted non-production environments in the ANZ Region. Monitoring confirms all Production environments have been restored and remain stable. We will continue to monitor Production and Non-Production environments to ensure no further issues. Next update will be provided by 9am AEST 4/2/2025.

9. resolved Feb 03, 2025, 10:07 PM UTC

   After considerable monitoring this incident is now resolved. We apologise for how you and your business may have been affected by this incident.

10. postmortem Feb 13, 2025, 10:27 PM UTC

    **Issue Summary** On 3 February 2025 a partial failure in the network occurred at 10.00am AEST. This impacted a subset of customers who then experienced a range of 5XX errors, and many users were unable to login. This disruption was between 10 to 15 minutes for the majority of customers, impacted on Production however the recovery continued until 2pm AEST for selected customers. The TechnologyOne teams first priority is to stabilise the Production environments, followed by recovery of non-production environments. TechnologyOne teams developed mitigations to recover from any re-occurrences of the issue until preventative actions could be established and implemented. TechnologyOne teams continued to work and analyse the root cause and develop actions through to 10/2/2025 Preventative actions began being applied progressively across regions from 2pm on 10/2/2025. **Root Cause** A partial failure on managed DNS caused network instability leading to intermittent access to application infrastructure. **Corrective Actions** Infrastructure was systematically reviewed and reset to recover from the hung state. Developed mitigations to recover for any re-occurrences. The managed DNS was recovered by our upstream vendor. **Preventative Actions** Implemented configuration changes in managed DNS based on recommendations from our upstream provider who reviewed the workload patterns to provide additional resilience to the service.