Is MindTouch down?

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of site degradation.

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of site degradation.

We are continuing to investigate this issue.

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of site degradation.

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of site degradation.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Mpower Expert Engineering team is investigating reports of site unavailability.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management - Fix Deployed - All Services Running Normally. The CXone Knowledge Management Engineering team has deployed a fix and all services are running normally. We are currently monitoring sites for deployment stability. Event duration 31 minutes

CXone Knowledge Management - Service Disruption Resolved - All Services Running Normally. The CXone Mpower Expert Engineering team has deployed a fix and monitored the deployment to make sure sites are stable. The issue is now resolved at this time. Event duration 38 minutes

**Impact Start Time \(UTC\) 07/09/2026 07:18 AM UTC** **Impact End Time \(UTC\) 07/09/2026 08:18 AM UTC** ## Incident Summary On 07/09/2026, some NiCE CXone Mpower customers experienced slow page load times, intermittent access issues, or temporary unavailability of the CXone Mpower Expert knowledge portal. The issue stemmed from a scheduled production release that inadvertently introduced a configuration change, reducing the resource allocation for the autoscaling service on the affected production cluster. The impact was resolved after the change was rolled back, redirecting traffic to the previous stable version and restoring service. ## Root Cause The issue stemmed from a scheduled production release that inadvertently introduced a configuration change, reducing the resource allocation for the autoscaling service on the affected production cluster. During the investigation, engineers determined that a previous resource increase unexpectedly failed to successfully merge into the codebase. As a result, the deployment reapplied the code-defined configuration, inadvertently overwriting the existing production setting and reverting the autoscaler to its previous resource allocation. This caused the autoscaler to encounter out-of-memory \(OOM\) conditions and was unable to scale cluster capacity as required, leading to degraded platform performance and temporary site unavailability for some customers. The release successfully passed standard staging validation and was deployed to another production environment without issue. The condition only manifested in the affected production cluster under higher traffic volume. While incidents of this nature are rare during routine production releases, the event highlighted opportunities to further strengthen deployment controls and configuration validation to reduce the risk of similar occurrences in the future ## Corrective Actions **Detection:** Internal monitoring and automated alerts identified the service failure, which was subsequently confirmed through customer reports of service degradation and temporary inaccessibility to some CXone Mpower Expert sites. Corrective Actions **Remediation:** The impact was resolved after the change was rolled back, redirecting traffic to the previous stable version and restoring service. This reduced cluster resource demand, allowing the autoscaling service to recover and resume normal operations. Completed on 07/09/2026. **Prevention:** The Engineering team restored the autoscaler's resource allocation in the affected production environment to its intended increased values, ensuring sufficient capacity to support cluster demand and reducing the risk of similar service impacts in the future. Completed on 07/10/2026. The release pipeline will be enhanced to introduce additional validation, and readiness checks earlier in the deployment process \("shift-left" testing\). This change will ensure that critical workloads, infrastructure capacity, and service health are fully validated before the traffic is routed to the new release version. An update will be provided by End of Day MT on 07/24/2026. NiCE resiliency teams remain focused on strengthening system reliability, stability, platform-wide resilience, and post-deployment validation, monitoring, and performance. These efforts are aligned under the Operational Resiliency Excellence initiative to drive sustained progress and measurable outcomes. ## Incident Timeline \(UTC\) 07/09/2026 07:18 AM \(UTC\) - Internal monitoring and deployment validation detected failures in the production environment. Engineering teams immediately began investigating the issue and initiated recovery actions. 07/09/2026 07:23 AM \(UTC\) - The first customer case was opened, and Tech Support \(TS\) engineers began their initial validation and troubleshooting investigation. 07/09/2026 07:56 AM \(UTC\) - While the remediation effort was ongoing, TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed. 07/09/2026 08:18 AM \(UTC\) - The affected release was successfully rolled back, restoring normal service. Following successful validation, the major incident was marked as resolved.

CXone Knowledge Management Service Degradation: Sites unavailable. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

CXone Knowledge Management Service Degradation: Sites unavailable. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

CXone Knowledge Management Service Degradation: Sites unavailable. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

CXone Knowledge Management Service Degradation: Sites unavailable. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

CXone Knowledge Management Service Degradation: Sites unavailable. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

CXone Knowledge Management Service Degradation: Sites unavailable. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

A fix has been implemented and we are monitoring the results.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management Service Degradation: Sites unavailable or slow to load. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management - Fix Deployed - All Services Running Normally. The CXone Knowledge Management Engineering team has deployed a fix and all services are running normally. We are currently monitoring sites for deployment stability. Event duration 3h 11m

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

A fix has been implemented and we are monitoring the results.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix has been released. Sites may still be experiencing degraded performance but will be coming fully back online shortly.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

**Impact Start Time \(UTC\) 07/01/2026 07:55 AM UTC** **Impact End Time \(UTC\) 07/01/2026 01:21 PM UTC** **Incident Summary:** On 07/01/2026, some NiCE CXone Mpower customers reported service degradation while accessing and using the CXone Mpower Expert knowledge portal. The recurring service degradation was caused by a Domain Name System \(DNS\) resolution bottleneck within the platform infrastructure during periods of elevated traffic. The impact was mitigated after engineers scaled up resources and implemented a node local DNS cache. Service stability was further improved as traffic volumes decreased, allowing the platform to recover and return to normal operation. ## Root Cause The recurring service degradation was caused by a DNS resolution bottleneck within the platform infrastructure during periods of elevated traffic. Increased DNS request volume exceeded infrastructure processing limits, resulting in intermittent DNS lookup failure. These failures triggered application retries, increased latency, elevated “5xx” errors, and customer-facing service degradation. The issue was traced to an existing configuration gap that prevented application workloads from utilizing the platform's DNS caching capability. As a result, DNS requests continued to follow the standard resolution path, creating unnecessary network overhead and contributing to DNS request saturation under higher traffic volumes. While this condition had existed for some time without widespread impact, continued platform growth, increased workload density, and infrastructure scaling changes increased DNS traffic and exposed the underlying limitation. Once the DNS processing limits were reached, requests were intermittently dropped, resulting in cascading application failures and customer-facing service disruptions. During the investigation, engineers implemented several mitigations to reduce customer impact while continuing to identify the underlying cause. However, because these actions addressed symptoms rather than the root issue, the condition recurred and resulted in multiple major incidents before a permanent corrective solution was implemented. ## Corrective Actions **Detection:** The failure condition triggered system alarms, prompting engineers to initiate incident response procedures to validate, investigate, and reproduce the issue. While the investigation was underway, some customers reported service degradation when accessing and using the CXone Mpower Expert knowledge portal. **Remediation:** The impact was mitigated after engineers scaled up resources and implemented a node local DNS cache. Service stability was further improved as traffic volumes decreased, allowing the platform to recover and return to normal operation. Completed on 07/01/2026. **Prevention:** Engineers developed a dashboard to monitor local DNS cache performance, improving visibility into DNS health and enabling earlier detection, investigation, and remediation of potential issues before they impact service availability or customer experience. Completed on 07/01/2026. The Engineering team implemented an emergency change to correctly enable local DNS caching for applicable workloads, reducing DNS traffic, preventing node-level DNS saturation, and restoring the intended DNS architecture. This improved platform stability and DNS performance while reducing the risk of similar incidents. Additional infrastructure optimizations were also implemented to improve operational efficiency and minimize service impact during scaling activities. Completed on 07/02/2026. Engineers will conduct a comprehensive validation of DNS configurations across all workloads to ensure consistent use of node local DNS cache. While the immediate issue has been remediated, a fleet-wide review will identify and correct any remaining configuration gaps, helping to optimize DNS performance, reduce infrastructure load, and prevent similar service degradation events in the future. In addition, the Engineering team will continue extended monitoring of platform health and stability to verify the long-term effectiveness of the corrective actions and proactively identify any emerging issues. An update will be provided by End of Day MT on 07/17/2026. NiCE resiliency teams remain focused on enhancing system reliability, stability, platform-wide resilience, monitoring capabilities, and overall performance. These ongoing efforts are aligned with the Operational Resiliency Excellence initiative, driving continuous improvement, strengthening operational maturity, and delivering measurable business outcomes. ## Incident Timeline \(UTC\) 07/01/2026 07:55 AM \(UTC\) - Engineers received a service degradation alert and immediately initiated incident response procedures to investigate, validate, and reproduce the issue to determine its scope and potential customer impact. 07/01/2026 08:15 AM \(UTC\) - The first customer case was opened, and Tech Support \(TS\) engineers began the troubleshooting investigation. 07/01/2026 08:39 AM \(UTC\) - TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed. 07/01/2026 09:54 AM \(UTC\) - Engineers identified a suspected cause and Incident Timeline \(UTC\) began remediation steps. 07/01/2026 10:31 AM \(UTC\) - A fix was implemented, and engineers closely monitored system performance. While the remediation reduced the impact, it did not fully restore service stability. Engineers continued their investigation and identification of further remediation actions. 07/01/2026 01:21 PM \(UTC\) - The impact was resolved after engineers scaled up resources and implemented a node local DNS cache, while traffic volume returned to normal levels. Following successful validations, the major incident was marked as resolved. Engineering teams continued extended monitoring of platform performance and kept the service disruption notification active on the Status page while validating the long-term effectiveness of the remediation. ‌ **Impact Start Time \(UTC\) 07/01/2026 02:11 PM UTC** **Impact End Time \(UTC\) 07/01/2026 07:18 PM UTC** **Incident Summary:** On 07/01/2026, some NiCE CXone Mpower customers reported service degradation while accessing and using the CXone Mpower Expert knowledge portal. The recurring service degradation was caused by a Domain Name System \(DNS\) resolution bottleneck within the platform infrastructure during periods of elevated traffic. The impact was mitigated after engineers scaled up cluster resources and updated application workloads to utilize node local DNS cache, reducing DNS traffic and preventing nodes from reaching DNS query processing limits. ## Root Cause The recurring service degradation was caused by a DNS resolution bottleneck within the platform infrastructure during periods of elevated traffic. Increased DNS request volume exceeded infrastructure processing limits, resulting in intermittent DNS lookup failure. These failures triggered application retries, increased latency, elevated “5xx” errors, and customer-facing service degradation. The issue was traced to an existing configuration gap that prevented application workloads from utilizing the platform's DNS caching capability. As a result, DNS requests continued to follow the standard resolution path, creating unnecessary network overhead and contributing to DNS request saturation under higher traffic volumes. While this condition had existed for some time without widespread impact, continued platform growth, increased workload density, and infrastructure scaling changes increased DNS traffic and exposed the underlying limitation. Once the DNS processing limits were reached, requests were intermittently dropped, resulting in cascading application failures and customer-facing service disruptions. During the investigation, engineers implemented several mitigations to reduce customer impact while continuing to identify the underlying cause. However, because these actions addressed symptoms rather than the root issue, the condition recurred and resulted in multiple major incidents before a permanent corrective solution was implemented. ## Corrective Actions **Detection:** The failure condition triggered system alarms, prompting engineers to initiate incident response procedures to validate, investigate, and reproduce the issue. While the investigation was underway, some customers reported service degradation when accessing and using the CXone Mpower Expert knowledge portal. **Remediation:** The impact was mitigated after engineers scaled up cluster resources and updated application workloads to utilize node local DNS cache, reducing DNS traffic and preventing nodes from reaching DNS query processing limits. These actions restored normal service behavior and improved overall platform stability. Completed on 07/01/2026. **Prevention:** Engineers developed a dashboard to monitor local DNS cache performance, improving visibility into DNS health and enabling earlier detection, investigation, and remediation of potential issues before they impact service availability or customer experience. Completed on 07/01/2026. The Engineering team implemented an emergency change to correctly enable local DNS caching for applicable workloads, reducing DNS traffic, preventing node-level DNS saturation, and restoring the intended DNS architecture. This improved platform stability and DNS performance while reducing the risk of similar incidents. Additional infrastructure optimizations were also implemented to improve operational efficiency and minimize service impact during scaling activities. Completed on 07/02/2026. Engineers will conduct a comprehensive validation of DNS configurations across all workloads to ensure consistent use of node local DNS cache. While the immediate issue has been remediated, a fleet-wide review will identify and correct any remaining configuration gaps, helping to optimize DNS performance, reduce infrastructure load, and prevent similar service degradation events in the future. In addition, the Engineering team will continue extended monitoring of platform health and stability to verify the long-term effectiveness of the corrective actions and proactively identify any emerging issues. An update will be provided by End of Day MT on 07/17/2026. NiCE resiliency teams remain focused on enhancing system reliability, stability, platform-wide resilience, monitoring capabilities, and overall performance. These ongoing efforts are aligned with the Operational Resiliency Excellence initiative, driving continuous improvement, strengthening operational maturity, and delivering measurable business outcomes. ## Incident Timeline \(UTC\) 07/01/2026 02:11 PM \(UTC\) - Engineers observed service degradation alert and the first customer case was opened. Engineers began their initial validation and troubleshooting investigation. 07/01/2026 02:55 PM \(UTC\) - TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed. 07/01/2026 03:14 PM \(UTC\) - Engineers identified a suspected cause and began remediation steps. 07/01/2026 03:49 PM \(UTC\) - Engineers began increasing cluster capacity Incident Timeline \(UTC\) and redistributed workloads to use the node local DNS cache. 07/01/2026 05:29 PM \(UTC\) - Sites became accessible but engineers were still seeing some “500” errors. Engineers planned to implement additional configuration changes to fully restore the service. 07/01/2026 05:55 PM \(UTC\) - Engineers continued resetting pods to stabilize and fully restore service. 07/01/2026 06:40 PM \(UTC\) - Problematic pods were removed from the affected nodes as part of ongoing recovery actions. 07/01/2026 07:18 PM \(UTC\) - The impact was resolved after engineers increased cluster resources and updated the application service with the node local DNS cache. Following successful validation, the major incident was marked as resolved. Since implementation of the fix, engineers have observed significant improvements in platform stability and continue to monitor performance to ensure sustained effectiveness.

CXone Knowledge Management Service Degradation: Sites are slow to load or inaccessible at this time. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

CXone Mpower Expert Service Degradation: Sites unavailable. The CXone Knowledge Management Engineering team is investigating reports of site unavailability.

CXone Knowledge Management Service Degradation: Sites unavailable. The issue has been identified and a fix is being worked on for deployment.

CXone Knowledge Management - Fix Deployed - All Services Running Normally. The CXone Knowledge Management Engineering team has deployed a fix and all services are running normally. We are currently monitoring sites for deployment stability. Event duration 31 mins

CXone Knowledge Management - Fix Deployed - All Services Running Normally. The CXone Knowledge Management Engineering team has deployed a fix and all services are running normally. We are currently monitoring sites for deployment stability. Event duration 31 mins

CXone Knowledge Management - Service Disruption Resolved - All Services Running Normally. The CXone Mpower Expert Engineering team has deployed a fix and monitored the deployment to make sure sites are stable. The issue is now resolved at this time. Event duration 31 mins

**Impact Start Time \(UTC\) 06/30/2026 02:07 PM UTC** **Impact End Time \(UTC\) 06/30/2026 02:57 PM UTC** **Incident Summary:** On 06/30/2026, some NiCE CXone Mpower customers reported service degradation while accessing and using the CXone Mpower Expert knowledge portal. The recurring service degradation was caused by a Domain Name System \(DNS\) resolution bottleneck within the platform infrastructure during periods of elevated traffic. The impact was mitigated by increasing pod capacity and provisioning additional healthy pods, which reduced the load on affected nodes and restored service availability. ## Root Cause The recurring service degradation was caused by a DNS resolution bottleneck within the platform infrastructure during periods of elevated traffic. Increased DNS request volume exceeded infrastructure processing limits, resulting in intermittent DNS lookup failure. These failures triggered application retries, increased latency, elevated “5xx” errors, and customer-facing service degradation. The issue was traced to an existing configuration gap that prevented application workloads from utilizing the platform's DNS caching capability. As a result, DNS requests continued to follow the standard resolution path, creating unnecessary network overhead and contributing to DNS request saturation under higher traffic volumes. While this condition had existed for some time without widespread impact, continued platform growth, increased workload density, and infrastructure scaling changes increased DNS traffic and exposed the underlying limitation. Once the DNS processing limits were reached, requests were intermittently dropped, resulting in cascading application failures and customer-facing service disruptions. During the investigation, engineers implemented several mitigations to reduce customer impact while continuing to identify the underlying cause. However, because these actions addressed symptoms rather than the root issue, the condition recurred and resulted in multiple major incidents before a permanent corrective solution was implemented. ## Corrective Actions **Detection:** Corrective Actions The failure condition triggered system alarms, prompting engineers to initiate incident response procedures to validate, investigate, and reproduce the issue. While the investigation was underway, some customers reported service degradation when accessing and using the CXone Mpower Expert knowledge portal. **Remediation:** The impact was mitigated by increasing pod capacity and provisioning additional healthy pods, which reduced the load on affected nodes and restored service availability. These new pods were not experiencing the DNS-related issues, allowing the impacted service to recover and return to normal operation. Completed on 06/30/2026. **Prevention:** Engineers developed a dashboard to monitor local DNS cache performance, improving visibility into DNS health and enabling earlier detection, investigation, and remediation of potential issues before they impact service availability or customer experience. Completed on 07/01/2026. The Engineering team implemented an emergency change to correctly enable local DNS caching for applicable workloads, reducing DNS traffic, preventing node-level DNS saturation, and restoring the intended DNS architecture. This improved platform stability and DNS performance while reducing the risk of similar incidents. Additional infrastructure optimizations were also implemented to improve operational efficiency and minimize service impact during scaling activities. Completed on 07/02/2026. Engineers will conduct a comprehensive validation of DNS configurations across all workloads to ensure consistent use of node local DNS cache. While the immediate issue has been remediated, a fleet-wide review will identify and correct any remaining configuration gaps, helping to optimize DNS performance, reduce infrastructure load, and prevent similar service degradation events in the future. In addition, the Engineering team will continue extended monitoring of platform health and stability to verify the long-term effectiveness of the corrective actions and proactively identify any emerging issues. An update will be provided by End of Day MT on 07/17/2026. NiCE resiliency teams remain focused on enhancing system reliability, stability, platform-wide resilience, monitoring capabilities, and overall performance. These ongoing efforts are aligned with the Operational Resiliency Excellence initiative, driving continuous improvement, strengthening operational maturity, and delivering measurable business outcomes. ## Incident Timeline \(UTC\) 6/30/2026 02:07 PM \(UTC\) - Engineers received a service degradation alert and immediately initiated incident response procedures to investigate, validate, and reproduce the issue to determine its scope and potential customer impact. 6/30/2026 02:11 PM \(UTC\) – The first customer case was opened, and Tech Support \(TS\) engineers began initial validation and troubleshooting investigation. 6/30/2026 02:30 PM \(UTC\) – TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed. 6/30/2026 02:42 PM \(UTC\) – Engineers identified a suspected cause and began remediation steps. 6/30/2026 02:57 PM \(UTC\) – The impact was mitigated after engineers increased pod capacity. After successful validation, the major incident was resolved. Engineers continued monitoring and root cause investigation.

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of degraded performance

We are continuing to investigate this issue.

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of degraded performance

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of degraded performance

CXone Knowledge Management Service Degradation. The CXone Knowledge Management Engineering team is investigating reports of degraded performance

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

**Impact Start Time \(UTC\) 06/29/2026 01:22 PM UTC** **Impact End Time \(UTC\) 06/29/2026 04:31 PM UTC** **Incident Summary:** On 06/29/2026, some NiCE CXone Mpower customers reported service degradation while accessing and using the CXone Mpower Expert knowledge portal. The recurring service degradation was caused by a Domain Name System \(DNS\) resolution bottleneck within the platform infrastructure during periods of elevated traffic. The impact was resolved after traffic volumes returned to normal levels. ## Root Cause The recurring service degradation was caused by a DNS resolution bottleneck within the platform infrastructure during periods of elevated traffic. Increased DNS request volume exceeded infrastructure processing limits, resulting in intermittent DNS lookup failure. These failures triggered application retries, increased latency, elevated “5xx” errors, and customer-facing service degradation. The issue was traced to an existing configuration gap that prevented application workloads from utilizing the platform's DNS caching capability. As a result, DNS requests continued to follow the standard resolution path, creating unnecessary network overhead and contributing to DNS request saturation under higher traffic volumes. While this condition had existed for some time without widespread impact, continued platform growth, increased workload density, and infrastructure scaling changes increased DNS traffic and exposed the underlying limitation. Once the DNS processing limits were reached, requests were intermittently dropped, resulting in cascading application failures and customer-facing service disruptions. During the investigation, engineers implemented several mitigations to reduce customer impact while continuing to identify the underlying cause. However, because these actions addressed symptoms rather than the root issue, the condition recurred and resulted in multiple major incidents before a permanent corrective solution was implemented. ## Corrective Actions **Detection:** Corrective Actions The failure condition triggered system alarms, prompting engineers to initiate incident response procedures to validate, investigate, and reproduce the issue. While the investigation was underway, some customers reported service degradation when accessing and using the CXone Mpower Expert knowledge portal. **Remediation:** The impact was resolved after traffic volumes returned to normal levels. Completed on 06/29/2026. **Prevention:** Engineers developed a dashboard to monitor local DNS cache performance, improving visibility into DNS health and enabling earlier detection, investigation, and remediation of potential issues before they impact service availability or customer experience. Completed on 07/01/2026. The Engineering team implemented an emergency change to correctly enable local DNS caching for applicable workloads, reducing DNS traffic, preventing node-level DNS saturation, and restoring the intended DNS architecture. This improved platform stability and DNS performance while reducing the risk of similar incidents. Additional infrastructure optimizations were also implemented to improve operational efficiency and minimize service impact during scaling activities. Completed on 07/02/2026. Engineers will conduct a comprehensive validation of DNS configurations across all workloads to ensure consistent use of node local DNS cache. While the immediate issue has been remediated, a fleet-wide review will identify and correct any remaining configuration gaps, helping to optimize DNS performance, reduce infrastructure load, and prevent similar service degradation events in the future. In addition, the Engineering team will continue extended monitoring of platform health and stability to verify the long-term effectiveness of the corrective actions and proactively identify any emerging issues. An update will be provided by End of Day MT on 07/17/2026. NiCE resiliency teams remain focused on enhancing system reliability, stability, platform-wide resilience, monitoring capabilities, and overall performance. These ongoing efforts are aligned with the Operational Resiliency Excellence initiative, driving continuous improvement, strengthening operational maturity, and delivering measurable business outcomes. ## Incident Timeline \(UTC\) 06/29/2026 01:22 PM \(UTC\) - Engineers received a service degradation alert and immediately initiated incident response procedures to investigate, validate, and reproduce the issue to determine its scope and potential customer impact. 06/29/2026 01:30 PM \(UTC\) - The first customer case was reported, and Technical Support \(TS\) engineers began validation and troubleshooting activities. 06/29/2026 03:34 PM \(UTC\) - TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed. 06/29/2026 03:39 PM \(UTC\) - Engineers identified a suspected contributing factor and initiated mitigation efforts to stabilize the service. 06/29/2026 04:31 PM \(UTC\) - Service performance recovered as traffic volumes returned to normal levels. Following successful validation of platform