MindTouch incident
CXone Mpower Expert - Service Degradation: US Sites unavailable
MindTouch experienced a major incident on December 31, 2025 affecting Application (General Service) and Search and 1 more component, lasting 7h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 31, 2025, 02:10 AM UTC
CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 02:29 AM UTC
CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 03:10 AM UTC
CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 03:30 AM UTC
CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- monitoring Dec 31, 2025, 03:34 AM UTC
A fix has been implemented and we are monitoring the results.
- investigating Dec 31, 2025, 03:52 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 04:16 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 04:32 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 04:51 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 05:06 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 05:23 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 05:38 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 05:53 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 06:11 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 06:25 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 06:41 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 06:56 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 07:11 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 07:53 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 08:24 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 08:42 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- investigating Dec 31, 2025, 08:59 AM UTC
CXone Mpower Expert Service Degradation: US Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.
- resolved Dec 31, 2025, 09:21 AM UTC
This incident has been resolved.
- postmortem Jan 07, 2026, 10:42 PM UTC
Impact Start Time \(UTC\) 12/31/2025 01:45 AM UTC Impact End Time \(UTC\) 12/31/2025 09:01 AM UTC **Incident Summary** Updated on 01/07/2026 - On 12/30/2025, some NiCE CXone Mpower customers reported experiencing intermittent service disruptions when utilizing the CXone Mpower Expert knowledge portal. The issue stemmed from a misconfigured network proxy, which prevented a critical cluster component from starting correctly on newly provisioned or restarted nodes. The impact was resolved after correcting the network proxy configuration and restarting the affected component pods, which fully restored service functionality. **Root Cause** The issue stemmed from a misconfigured network proxy, which prevented a critical cluster component from starting correctly on newly provisioned or restarted nodes. This led to node-specific failures in service routing and Domain Name System \(DNS\) resolution. Because the network proxy is responsible for programming node-level routing, the affected nodes were unable to reliably resolve or route traffic. The impact was limited to specific nodes and only affected workloads running on them, which accounts for the intermittent nature of the failures. Engineers later determined that the misconfiguration was introduced after enabling a component for enhanced logging due to an oversight. ## Corrective Actions **Detection:** Internal support teams received reports from customers indicating intermittent service disruptions when utilizing the CXone Mpower Expert knowledge portal. **Remediation:** The impact was resolved after correcting the network proxy configuration and restarting the affected component pods, which fully restored service functionality. Completed on 12/31/2025. **Prevention:** The Engineering team will implement enhanced monitoring and alerting for network proxy crash loops to proactively detect similar issues and enable immediate remediation, preventing customer impact. An update will be provided by End of Day \(EOD\) MT on 01/23/2026. The Engineering team is implementing an automated validation process for network proxy configuration changes to detect issues early and prevent future service disruptions. An update will be provided by EOD MT on 01/30/2026. **Incident Timeline \(UTC\)** 12/31/2025 01:45 AM \(UTC\) - The first customer case was opened, and Tech Support \(TS\) engineers began their initial validations troubleshooting investigation. 12/31/2025 02:31 AM \(UTC\) - TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed. 12/31/2025 03:00 AM \(UTC\) - Engineers observed that sites were restored but were still experiencing latency. 12/31/2025 04:40 AM \(UTC\) - Engineers restarted the pods to allow the system to create new pods. 12/31/2025 04:54 AM \(UTC\) - The affected sites were working fine but engineers were still seeing proxy errors on the backend. 12/31/2025 05:22 AM \(UTC\) - The proxy errors were still present, and engineers found DNS failures. Investigation and troubleshooting continued. 12/31/2025 06:09 AM \(UTC\) - Engineers restarted the deploy and infrastructure pipelines. 12/31/2025 08:00 AM \(UTC\) - Engineers reported that the automated deployment of pipelines failed and will need to perform manual deployment. 12/31/2025 09:01 AM \(UTC\) - The impact was resolved after engineers corrected the proxy configuration and restarted the pods. Following successful test validations, the major incident was marked as resolved.