TechnologyOne incident

Access Issues - ANZ Region (excluding ANZ Federal Government)

Major Resolved View vendor source →

TechnologyOne experienced a major incident on May 18, 2024 affecting Database Services and Ci in the Cloud, lasting 17h 27m. The incident has been resolved; the full update timeline is below.

Started
May 18, 2024, 10:35 AM UTC
Resolved
May 19, 2024, 04:03 AM UTC
Duration
17h 27m
Detected by Pingoru
May 18, 2024, 10:35 AM UTC

Affected components

Database ServicesCi in the Cloud

Update timeline

  1. investigating May 18, 2024, 10:35 AM UTC

    We have identified a selection of customer environments experiencing issues accessing services in CiAnywhere. Our engineers are currently investigating the root cause.

  2. identified May 18, 2024, 10:37 AM UTC

    We have identified the root cause and our engineers are working with our upstream provider to resolve the issue.

  3. identified May 18, 2024, 12:13 PM UTC

    We have started recovering services.

  4. monitoring May 18, 2024, 01:39 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved May 19, 2024, 04:03 AM UTC

    This incident has been resolved.

  6. postmortem Jun 13, 2024, 04:33 AM UTC

    Service Disruption: On May 18, 2024 evening, customers may have experienced a disruption accessing their SaaS environments. Root Cause Analysis: A change in operating parameters on the Directory service triggered an automatic recovery process that, as a side effect, caused a secondary problem. Which in turn affected the ability to do Directory service lookup, which caused the interruption in service. Corrective Actions: We worked with our upstream provider to diagnose the issue, and resolve the problem, restoring full service availability. Preventive Measures: We have begun an in-depth architectural review of the service to better secure it against disruption. Improving the resiliency of our services is an ongoing priority for us.