Orbus Software experienced a major incident on February 18, 2025 affecting OrbusInfinity and OrbusInfinity and 1 more component, lasting 1d 14h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 18, 2025, 10:39 PM UTC
We are aware of an incident preventing users from accessing OrbusInfinity. Engineers are engaged and working to remediate the issue. Updates will be provided in due course.
- investigating Feb 18, 2025, 10:51 PM UTC
We are continuing to investigate this issue.
- identified Feb 19, 2025, 04:00 AM UTC
Access to OrbusInfinity should now be restored. Users may need to clear their browser cache to access the application. There may still be intermittent access issues for Draw, Dashboards and the Home Page. Engineers are engaged and working to remediate these issues. Further updates will be provided in due course.
- identified Feb 19, 2025, 09:27 AM UTC
Engineers continue to investigate issues directly related to the availability if the OrbusInfinity Home Page, Dashboards and Draw functions. If you are experiencing issues in other areas of the application, please clear your local browser cache and try again. Further updates will be provided as they become available.
- identified Feb 19, 2025, 05:21 PM UTC
Engineers continue to work towards a resolution. The root cause of the issue has been clearly identified and activities to fix are ongoing. Further updates will be provided at 20:30 GMT (07:30 AEDT / 00:30 GST)
- monitoring Feb 19, 2025, 08:26 PM UTC
A fix has been implemented and full functionality is restored in all impacted regions. Engineers will continue to monitor the platform for a period before the incident is marked as resolved.
- resolved Feb 20, 2025, 01:32 PM UTC
Following the restoration of all functionality, normal service levels and a stable period of monitoring, the incident has been marked as resolved.
- postmortem Mar 05, 2025, 03:56 PM UTC
# **Incident Summary** On 18/02/2025 At 21:41 UTC it was reported that some users were unable to access OrbusInfinity across the Australia East, IRAP and UAE North regions. Users were getting inconsistent results when trying to navigate the Content Area of the platform and load specific application pages. Often a "Something Went Wrong" error was displayed to the user. The issue could be partially mitigated by clearing local browser caches. Following a manual clearing of a browser cache, just the Home Page, Dashboards and Draw functions of the application remained in a non-functional state. Engineers implemented a full fix at 18:56 UTC on 19/02/2025 with all OrbusInfinity functionality restored in all impacted regions. # **Impact** OrbusInfinity pages loaded inconsistently with a "Something Went Wrong" error being displayed to end-users. Following a clear down of local browser caches by end-users, just the Home Page, Dashboards and Draw functions in OrbusInfinity were impacted. # **Root Cause Analysis** On 18/02/2025 version 1.123.2 of the OrbusInfinity application was deployed to all regions. This was successful in all regions with no exceptions or errors noted by operational monitoring systems. Following the update in the Australia East, IRAP and UAE North regions, periodic "Something Went Wrong" errors were reported by end users. Following investigation, it was discovered that the OrbusInfinity cloud provider's Content Delivery Networks \(CDN\) had an issue which resulted in the CDN failing to correctly clear its cache following the release. This resulted in different, incompatible versions of application files being served to users. The erroneous behavior corrected itself with no intervention from Orbus engineers in the IRAP and UAE North regions. The CDN cache expired for these regions and the newly released versions of the files read in. In the Australia East region, Orbus Engineers deployed a hotfix which guaranteed the latest version of the application would be served to users, even if the CDN held on to an older version of the application files. The workaround has since been formalised and rolled out to all other regions. Although OrbusInfinity will no longer be affected by the underlying problem, a support ticket remains open with the cloud provider.