Performance Degradation - CiAnywhere / ANZ Region

Update timeline

1. investigating May 01, 2024, 02:02 AM UTC

   We have identified a selection of customer environments experiencing performance degradation within ANZ. Our engineers are currently investigating the root cause. We will provide another update within the next 60 minutes. Thank you.

2. identified May 01, 2024, 02:48 AM UTC

   We have identified the cause of the issue and are working towards developing and implementing the required changes to restore the performance back to standard operating levels. We shall provide an update in the next 60 minutes.

3. monitoring May 01, 2024, 05:02 AM UTC

   We apologise for the delayed update. We are pleased to inform you that we have applied the change and have obtained feedback that the change has been successful, where performance have started to improve. We will continue to monitor this before resolving it.

4. resolved May 01, 2024, 11:14 PM UTC

   Monitoring throughout the day has proven the actions taken yesterday were successful. We will follow up with root cause and post mortem in the next 14 days. Thank you for your continued support and feedback

5. postmortem May 16, 2024, 02:13 AM UTC

   **Issue Summary:** On Wednesday 1 May 2024 customers experienced performance degradation within Ci Anywhere including connection errors, general lagging in navigation from 11am. ‌ **Root Cause Analysis:** A software change included in Fastline week 17 caused the background service used to cache records to increase the connections beyond the maximum limits. The testing undertaken for the change did not appropriately cover the load testing. ‌ **Corrective Measures:** Additional services for background caching were built and across the services with prioritisation given to production environments. Snapshots were adjusted as follows: * Mainline snapshot for May was updated to week prior \(week 16\) where this software change was not present. * Fastline snapshot for Week 18 was updated to remove the changes from week 17. ‌ **Preventive Measures:** Testing for software changes relating to caching has been improved to include load/pressure tests.