TechnologyOne incident

Performance Degradation - ANZ Region / 2024A Release

Minor Resolved View vendor source →

TechnologyOne experienced a minor incident on May 14, 2024, lasting 12h 25m. The incident has been resolved; the full update timeline is below.

Started
May 14, 2024, 02:39 AM UTC
Resolved
May 14, 2024, 03:04 PM UTC
Duration
12h 25m
Detected by Pingoru
May 14, 2024, 02:39 AM UTC

Update timeline

  1. investigating May 14, 2024, 02:39 AM UTC

    We have identified a selection of customer environments experiencing performance degradation within ANZ. Our engineers are currently investigating the root cause. We will provide another update within the next 60 minutes. Thank you.

  2. investigating May 14, 2024, 03:30 AM UTC

    We are continuing to investigate the issue and will provide a further update within the next 60 minutes.

  3. investigating May 14, 2024, 04:24 AM UTC

    We are continuing to investigate this issue and our investigation shows this issue is isolated to parts of cia. We will provide a further update within the next 60 minutes. We apologise for the inconvenience this issue is causing.

  4. investigating May 14, 2024, 05:16 AM UTC

    Our investigation continues however we have found using Private/Incognito browser is allowing navigation and BAU activities to proceed. We will provide a further update within 60 minutes. We apologise for the inconvenience this issue is causing.

  5. monitoring May 14, 2024, 06:09 AM UTC

    We have applied a change to restore functionality. Users should log out and log back in and undertake a browser a refresh: - Ctrl + F5 (PC Users) - Cmd + Shift + R (Mac Users) We will monitor for a period of time before setting to resolved.

  6. resolved May 14, 2024, 03:04 PM UTC

    This incident has been resolved.

  7. postmortem May 28, 2024, 02:43 AM UTC

    **Issue Summary** On Tuesday 14 May 2024 at approximately 10.30am AEST customers started experiencing connection errors and slow response times within cia on the 2024A release. At 3.30pm AEST TechnologyOne implemented a fix and due to the issue found that users were required to perform a hard refresh of their browser or use incognito/private browsing in addition to the fix implemented. **Root Cause Analysis** A feature was enabled for Production environments on 2024A. **Corrective Measures:** Disabled the feature for Production environments on 2024A. Recycled App Servers in ANZ to flush server caches. Informed customers to perform hard refresh on browser or use incognito/private browsing. **Preventive Measures:** The process for feature enablement has been updated and circulated to all teams.