Service Disruption for a subset of customers / ANZ Region / 2025A Releases

Update timeline

1. investigating Apr 03, 2025, 11:51 PM UTC

   We are investigating an issue impacting service for ANZ Region / 2025A. Impact/Error/How to verify: A subset of customers in ANZ Region for 25A Releases are experiencing 500 and 502 errors intermittently in CiAnywhere. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

2. monitoring Apr 04, 2025, 12:22 AM UTC

   Our team has verified the implementation of a fix is complete for ANZ Region / 25A Releases. Please refresh/clear your browser cache or try a private/incognito browser if you still receive an error. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

3. investigating Apr 04, 2025, 01:08 AM UTC

   We have seen new errors arise since fix applied. We are investigating this further. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

4. investigating Apr 04, 2025, 01:37 AM UTC

   Our team is continuing to investigate. Impact/Error/How to verify: A subset of customers in ANZ Region for 25A Releases are experiencing 502 errors or Service Offline error intermittently in CiAnywhere. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

5. investigating Apr 04, 2025, 02:07 AM UTC

   Our team is continuing to investigate the issue, and our investigation is narrowing down. Impact/Error/How to verify: A subset of customers in ANZ Region for 2025A Releases are experiencing 502 errors or Service Offline error intermittently in CiAnywhere. Our investigation shows this is occurring for Financials and Supply Chain processors. Due to the investigation, the next update will be provided in 30 minutes, or sooner if new information becomes available.

6. identified Apr 04, 2025, 02:35 AM UTC

   Our team has identified the issue and have applied mitigations to reduce the errors received. We continue to work on a complete fix. The next update will be provided in 60 minutes, or sooner if new information becomes available.

7. identified Apr 04, 2025, 03:35 AM UTC

   Our logs show that errors do continue however at a reduced rate. We continue to work on a complete fix. The next update will be provided in 60 minutes, or sooner if new information becomes available.

8. identified Apr 04, 2025, 04:34 AM UTC

   Our logs show that errors are minimal right now. We continue to work on a fix for the underlying issue. The next update will be provided in 60 minutes, or sooner if new information becomes available.

9. identified Apr 04, 2025, 05:36 AM UTC

   Our logs show that errors are very limited now. We continue to work on a fix for the underlying issue. The next update will be provided in 60 minutes, or sooner if new information becomes available.

10. monitoring Apr 04, 2025, 05:43 AM UTC

    Our team has verified the implementation of a fix is complete. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.

11. resolved Apr 04, 2025, 07:43 AM UTC

    After 2 hours monitoring, this incident is now resolved. We will perform a post incident review and post here on completion. We apologise for how you and your business may have been affected by this incident.

12. postmortem Apr 22, 2025, 06:54 AM UTC

    **Issue Summary** On 4 April 2025 a subset of customers experienced intermittent 502 and server unavailable errors from 8.32am AEST through to 1.35pm AEST. Initial analysis pointed to unhealthy servers causing the 502 and server unavailable errors. Impacted customer datasets were moved to a healthy server stack. The errors began to appear on the new server stack. Further analysis found a software issue causing a stack overflow when a user on a specific dataset performed a specific action. This dataset was isolated to ensure no further disruption to other customer datasets. **Root Cause** Stack overflow exception triggered by a specific workflow activity within the Data Entry app. **Corrective Actions** Replaced unhealthy servers with new servers and moved customer datasets to the healthy servers. Isolated customer dataset triggering the stack overflow exception. **Preventative Actions** Observability enhanced to identify any stack overflow exceptions and triggering application including an alert. Resolve underlying software issue to address the stack overflow exception.