TechnologyOne incident

UK - Service Degradation for Timetabling & Scheduling Non Production

Minor Resolved View vendor source →

TechnologyOne experienced a minor incident on November 4, 2024, lasting 3h 49m. The incident has been resolved; the full update timeline is below.

Started
Nov 04, 2024, 10:43 AM UTC
Resolved
Nov 04, 2024, 02:33 PM UTC
Duration
3h 49m
Detected by Pingoru
Nov 04, 2024, 10:43 AM UTC

Update timeline

  1. identified Nov 04, 2024, 10:43 AM UTC

    Our team has identified a fix which is being implemented now for Timetabling & Scheduling Non Production environments in the UK. The bulk of customer environments are resolved, with work ongoing to correct the remaining systems.

  2. monitoring Nov 04, 2024, 11:31 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring Nov 04, 2024, 11:31 AM UTC

    We are continuing to monitor for any further issues.

  4. resolved Nov 04, 2024, 02:33 PM UTC

    This incident has been resolved.

  5. postmortem Nov 18, 2024, 11:27 PM UTC

    **Issue Summary:** On Monday 04 Nov 2024 at 9:50 it was noted that customers experienced performance degradation within non-production environments for T&S applications. The TechnologyOne team began an investigation immediately and quickly identified a fix. **Root Cause Analysis:** Servers found to be in an error state due to upstream maintenance which had an adverse affect on running services for the non-production environments. **Corrective Measures:** Manual intervention was required to failover services to return normal service. **Preventive Measures:** Enhancements have been identified to increase the resilience of running services. Additional proactive alerting has been created to attempt to predict the issues recurring in the future.