TechnologyOne incident
Unable to access Ci for a subset of customers / ANZ Region / 2023B
TechnologyOne experienced a critical incident on July 15, 2024 affecting Ci in the Cloud, lasting 4h 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 15, 2024, 11:32 PM UTC
Our team is troubleshooting an issue impacting Ci Access for ANZ Region across 2023B customers. Customers impacted by this incident would be presented with a 502 or 504 Bad Gateway Error Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.
- identified Jul 16, 2024, 12:27 AM UTC
Our team has identified a fix which is being implemented now for ANZ Region / 2023B We anticipate the implementation of the fix to take 30 minutes to complete and verify. Next update shall be provided in 60 minutes or sooner if new information becomes available.
- monitoring Jul 16, 2024, 12:56 AM UTC
Our team has verified the implementation of a fix is complete for ANZ Region across 2023B customers. We will monitor the logs until the end of business day before resolving this incident.
- resolved Jul 16, 2024, 04:30 AM UTC
After 3 hours monitoring this incident is now resolved. We will undertake a Post Incident Review and findings will be posted here on completion. We apologise for how you and your business may have been affected by this incident.
- postmortem Sep 05, 2024, 06:19 AM UTC
## Issue Summary On 17/06/2024, multiple customers were presented with Error 502 and 504 when their users attempted to load into CI in the cloud \(Production\). ## Root Cause Analysis A configuration error in scaling values during routine maintenance. ## Corrective Actions Scaling values were increased to accommodate customers' size and scale, with servers updated to adopt these values. ## Preventive Measures Standard operating procedures have been updated to ensure correct configuration is set for scaling moving forward.