CXone Mpower Expert - Service Degradation: Sites unavailable US

Update timeline

1. monitoring Oct 10, 2025, 03:19 AM UTC

   CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.

2. resolved Oct 10, 2025, 03:37 AM UTC

   This incident has been resolved.

3. postmortem Oct 20, 2025, 06:00 PM UTC

   Impact Start Time \(UTC\) 10/10/2025 03:03 AM UTC Impact End Time \(UTC\) 10/10/2025 03:37 AM UTC **Incident Summary** Updated on 10/17/2025 - On 10/10/2025, some NiCE CXone Mpower customers reported an inability to access the CXone Mpower Expert knowledge portal, encountering a “504 Gateway Time-out” error on all pages. The issue stemmed from a scheduled, routine change release intended to provision new resources, which resulted in a mismatch between release versions and caused an interruption in traffic routing. The impact was resolved by correcting the release version on the affected application load balancer, which successfully restored the impacted production sites. **Root Cause** The issue stemmed from a scheduled, routine change release intended to provision new resources, which resulted in a mismatch between release versions and caused an interruption in traffic routing. During the deployment, manual intervention was required, which resulted in the removal of the older version while migrating to the new one. As traffic was transitioned, sites experienced downtime due to improper traffic draining, since the removal procedure was not aligned with the recommended automated process. The issue was ultimately traced to a flaw in the deployment process used as the software release strategy for one of the load balancer components from a previous release. Although the problem was quickly identified and promptly corrected during deployment, engineers recognized key opportunities to improve the deployment process further to prevent similar occurrences in the future **Corrective Actions** **Detection:** Internal support teams detected a potentially customer-impacting issue during a change deployment, which was later confirmed by customer reports indicating an inability to access the CXone Mpower Expert knowledge portal, encountering a “504 Gateway Time-out” error on all pages. **Remediation:** The impact was resolved by correcting the release version on the affected application load balancer, which successfully restored the impacted production sites. Completed on 10/10/2025. **Prevention:** The Engineering team will enhance the deployment process by upgrading it to ensure that each version uses its own instance. This improvement will allow versions to operate independently and prevent similar issues in the future. Furthermore, testing procedures will be strengthened to include both old and new release versions during the Staging phase. An update will be provided by End of Day \(EOD\) MT on 10/31/2025. The Engineering team is updating the deployment documentation and automation scripts to prevent residual deployments from interfering with traffic routing and to avoid similar incidents in the future. Furthermore, an additional validation stage will be added to the release process to verify the previous release and ensure configuration consistency before deployment. An update will be provided by EOD MT on 11/14/2025. **Incident Timeline \(UTC\)** 10/10/2025 03:03 AM \(UTC\) - Engineers identified a potentially customer-impacting issue during a change deployment. 10/10/2025 03:08 AM \(UTC\) - Engineers notified the Network Operations Center \(NOC\) engineers about a possible customer impact during a change deployment. 10/10/2025 03:16 AM \(UTC\) - The first customer case was opened, and Tech Support \(TS\) engineers initiated the troubleshooting investigation. This case was later confirmed to be related to the identified problem and subsequently linked to the major incident. 10/10/2025 03:24 AM \(UTC\) - A major incident was raised and confirmed due to a potential impact on customer services. 10/10/2025 03:37 AM \(UTC\) - Engineers rectified the configuration issues that caused the impact. Internal validation confirmed successful restoration of services, and the major incident was then marked as resolved.