MindTouch incident

CXone Mpower Expert - Service Degradation for Generative Search. Status = identified

Major Resolved View vendor source →

MindTouch experienced a major incident on November 7, 2025 affecting Generative Search, lasting 2h 22m. The incident has been resolved; the full update timeline is below.

Started
Nov 07, 2025, 03:21 PM UTC
Resolved
Nov 07, 2025, 05:43 PM UTC
Duration
2h 22m
Detected by Pingoru
Nov 07, 2025, 03:21 PM UTC

Affected components

Generative Search

Update timeline

  1. investigating Nov 07, 2025, 03:21 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The CXone Mpower Expert Engineering team is investigating reports of Generative Search not working.

  2. investigating Nov 07, 2025, 03:40 PM UTC

    CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of Generative Search being unavailable.

  3. identified Nov 07, 2025, 03:42 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  4. identified Nov 07, 2025, 04:00 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  5. identified Nov 07, 2025, 04:15 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  6. identified Nov 07, 2025, 04:30 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  7. identified Nov 07, 2025, 04:45 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  8. identified Nov 07, 2025, 05:00 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  9. identified Nov 07, 2025, 05:15 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  10. identified Nov 07, 2025, 05:40 PM UTC

    CXone Mpower Expert Service Degradation: Generative Search Unavailable. The issue has been identified and a fix is being worked on for deployment.

  11. resolved Nov 07, 2025, 05:43 PM UTC

    This incident has been resolved.

  12. postmortem Mar 16, 2026, 11:12 PM UTC

    **Impact Start Time \(UTC\):** 11/07/2025 03:05 PM UTC **Impact End Time \(UTC\):** 11/08/2025 05:43 AM UTC ‌ **Incident Summary:** Updated on 01/23/2026 - On 11/08/2025, a NiCE CXone Mpower customer reported that the Generative Search feature was failing to return any results within the CXone Mpower Expert knowledge portal. The issue stemmed from a period of heavy data ingestion, which led to high Central Processing Unit \(CPU\) utilization and a sharp increase in throttling on our third-party vendor’s database. The issue was resolved by scaling up CPU resources within our platform, allowing it to manage the increased utilization and restore affected services. ‌ **Root Cause:** The issue stemmed from a period of heavy data ingestion, which led to high CPU utilization and a sharp increase in throttling on our third-party vendor’s database. Monitoring metrics confirmed that the cluster was under substantial load due to intensive upsert and indexing operations. This activity temporarily saturated CPU resources, causing degraded performance and intermittent timeouts. Further investigation revealed that a large dataset was likely uploaded without batching, creating a concentrated indexing workload across all nodes. Additionally, some nodes restarted during the scaling process, adding to the temporary instability. These factors collectively resulted in severe latency and timeouts in the production cluster, leading to the customer-reported impact. ‌ **Corrective Actions:** _Detection_: While engineers were investigating latency and timeout issues on the platform, a customer reported that the Generative Search feature was not returning any results within the CXone Mpower Expert knowledge portal. _Remediation_: The issue was resolved by scaling up CPU resources within our platform, allowing it to manage the increased utilization and restore affected services. The vendor later confirmed that once the ingestion and indexing processes were completed, the cluster stabilized, and performance returned to normal. Completed on 11/07/2025. _Prevention_: Our Engineering team, in collaboration with our third-party vendor, will be implementing a new feature that provides the platform with a dedicated indexing cluster. This enhancement will improve data ingestion performance and help prevent similar incidents in the future. An update will be provided by End of Day MT on 03/20/2026. ‌ **Incident Timeline \(UTC\):** 11/07/2025 03:05 PM \(UTC\) - The customer case was opened, and Tech Support \(TS\) engineers initiated validation and troubleshooting. This was later confirmed to be related to the ongoing platform issue. 11/07/2025 03:19 PM \(UTC\) - After identifying potential impact on the production cluster, they notified the Network Operations Center \(NOC\) engineers. service outage notification was published on the Status portal to inform customers of the issue and its expected impact. 11/07/2025 03:22 PM \(UTC\) - A major incident was proposed and confirmed. 11/07/2025 03:42 PM \(UTC\) - Engineers, in collaboration with the third-party vendor, identified a probable root cause and initiated remediation efforts. 11/07/2025 05:43 PM \(UTC\) - The impact was resolved after engineers increased the capacity of CPU resources. Following successful test validations, the major incident was marked as resolved.