CXone Mpower Expert - Service Degradation: Loading issues

Update timeline

1. investigating Oct 03, 2025, 06:23 AM UTC

   CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.

2. investigating Oct 03, 2025, 06:26 AM UTC

   We are continuing to investigate this issue.

3. investigating Oct 03, 2025, 06:34 AM UTC

   We are continuing to investigate this issue.

4. resolved Oct 03, 2025, 06:35 AM UTC

   This incident has been resolved.

5. postmortem Oct 09, 2025, 04:45 PM UTC

   **Incident Summary** Updated on 10/08/2025 - On 10/03/2025, some NiCE CXone Mpower customers reported that articles, directories, and thumbnails were not working, and the search page was failing to load on the CXone Mpower Expert knowledge portal. The issue stemmed from a recent change release implemented to spin up new resources and add new configuration routing rules, which introduced a mismatch in traffic routing between the new and old release versions. The impact was resolved by migrating all sites to the latest release version, which immediately restored all affected services. **Root Cause** The issue stemmed from a recent change release implemented to spin up new resources and add new configuration routing rules, which introduced a mismatch in traffic routing between the new and old release versions. This was caused by a flaw in the deployment process used as a software release strategy on one of the load balancer components. During this process, both versions were configured to share the same load balancer component instead of operating on separate instances. This shared configuration created a mismatch between the release versions, causing the impact. After the new routing rule was added, traffic was directed correctly to the new release version, but it didn’t work on the old version because that version didn’t have the new rule. Although testing was performed in the lower environment during the Staging phase, the problem went undetected until the deployment reached the production environment. Following this, engineers identified the need to enhance the testing process to include both the old and new release versions, ensuring similar issues are prevented in the future. **Corrective Actions** **Detection:** Internal support teams received reports from customers indicating that articles, directories, and thumbnails were not working, and the search page was failing to load on the CXone Mpower Expert knowledge portal. **Remediation:** The impact was resolved by migrating all sites to the latest release version, which immediately restored all affected services. Completed on 10/03/2025. **Prevention:** Although no similar change releases are scheduled in the near future to mitigate the risk of recurrence, the Engineering team will enhance the deployment process by upgrading it to ensure that each version uses its own instance. This improvement will allow versions to operate independently and prevent similar issues in the future. Furthermore, testing procedures will be strengthened to include both old and new release versions during the Staging phase. An update will be provided by End of Day \(EOD\) MT on 10/31/2025. The Engineering team will review the existing alarm mechanisms to evaluate whether adjustments are needed and explore ways to make monitoring more granular. These improvements will enhance proactive detection, enabling faster issue resolution and preventing potential impact on customers. An update will be provided by EOD MT on 10/31/2025. ‌ **Incident Timeline \(UTC\)** 10/03/2025 05:50 AM \(UTC\) - The first customer case was opened, and Tech Support \(TS\) engineers began their initial validations and troubleshooting investigation. 10/03/2025 06:19 AM \(UTC\) - While engineers were already aware of the issue and had identified a suspected cause, they immediately began their remediation efforts. 10/03/2025 06:21 AM \(UTC\) - TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed. 10/03/2025 06:35 AM \(UTC\) - The impact was resolved after engineers migrated all sites to the new release version. Following successful validations, the major incident was marked as resolved.