MindTouch Outage History

Started Mar 20, 2025, 01:46 PM UTC· resolved Mar 20, 2025, 04:27 PM UTC

1. investigating Mar 20, 2025, 01:46 PM UTC

   CXOne MPower Expert Service Degradation: Sites unavailable. The CXOne MPower Expert Engineering team is investigating reports of site unavailability.

2. investigating Mar 20, 2025, 01:54 PM UTC

   We are continuing to investigate this issue.

3. monitoring Mar 20, 2025, 02:01 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Mar 20, 2025, 02:17 PM UTC

   We are continuing to monitor for any further issues.

5. monitoring Mar 20, 2025, 02:33 PM UTC

   We are continuing to monitor for any further issues.

6. monitoring Mar 20, 2025, 02:48 PM UTC

   We are continuing to monitor for any further issues.

7. monitoring Mar 20, 2025, 03:03 PM UTC

   We are continuing to monitor for any further issues.

8. monitoring Mar 20, 2025, 03:19 PM UTC

   We are continuing to monitor for any further issues.

9. monitoring Mar 20, 2025, 03:34 PM UTC

   We are continuing to monitor for any further issues.

10. monitoring Mar 20, 2025, 04:11 PM UTC

    We are continuing to monitor for any further issues.

11. monitoring Mar 20, 2025, 04:26 PM UTC

    We are continuing to monitor for any further issues.

12. resolved Mar 20, 2025, 04:27 PM UTC

    Expert Monitoring Complete - services running normally

13. postmortem Mar 26, 2025, 06:54 PM UTC

    **Impact Start Time \(UTC\) 03/20/2025 01:46 PM UTC** **Impact End Time \(UTC\) 03/20/2025 02:01 PM UTC** **Incident Summary** Updated on 03/26/2025 - On 03/20/2025, some CXone Mpower customers reported being unable to access the CXone Mpower Expert knowledge portal, encountering "503" and "504" error messages. The issue occurred during the regular Quality Assurance \(QA\) site creation process, where rapidly deploying multiple sites through a system script generated an unexpectedly high load, placing excessive strain on infrastructure components. The impact was resolved by restarting the affected services and performing a rolling restart on the affected nodes, restoring services to normal operation. **Root Cause** The issue occurred during the regular QA site creation process, where rapidly deploying multiple sites through a system script generated an unexpectedly high load, placing excessive strain on infrastructure components. As part of a regular deployment process, several QA sites were created to run integration tests before directing customer traffic to the new deployment. However, this unexpectedly caused frequent reloads of the load balancer, leading to timeouts and unresponsive pages. Ultimately, this triggered alerts and resulted in customer impact. Although this procedure had never previously caused such an issue, engineers recognized the need to enhance the system to accommodate the growing load in the production environment driven by an increasing number of customers and their utilization. They promptly developed and implemented preventive measures to mitigate the risk of similar incidents in the future. **Corrective Actions** **Detection:** Internal support teams detected a potentially customer-impacting issue through proactive alarms and monitoring mechanisms, which was later confirmed by customer reports being unable to access the CXone Mpower Expert knowledge portal, encountering "503" and "504" error messages. **Remediation:** The impact was resolved by restarting the affected services and performing a rolling restart on the affected nodes, restoring services to normal operation. Completed on 03/20/2025. **Prevention:** Engineering team implemented rate-limiting measures to control the number of QA sites created simultaneously and increased the pause duration between each site's creation, preventing excessive load spikes during such procedure. Completed on 03/20/2025 **Risk of Reoccurrence of Impact:** Low **Incident Timeline \(UTC\)** 03/20/2025 01:46 PM \(UTC\) - Internal support teams received potentially customer-impacting alerts and posted a service disruption notification on the Status Health Portal. Simultaneously, the first customer case was opened, prompting Tech Support \(TS\) engineers to begin their initial validation and troubleshooting investigation, which later confirmed the issue was related to the major incident. 03/20/2025 01:48 PM \(UTC\) - Engineers proactively raised a major incident while continuing to work on restoring the service. 03/20/2025 01:51 PM \(UTC\) - Engineers restarted the affected service components, stabilizing the system. They continued to monitor the system’s health. 03/20/2025 02:01 PM \(UTC\) - After further monitoring and health checks, the impact was confirmed to be fully resolved. Following successful test validations, the major incident was officially marked as resolved

Read the full incident report →

Started Mar 13, 2025, 07:42 PM UTC· resolved Mar 13, 2025, 08:41 PM UTC

1. investigating Mar 13, 2025, 07:42 PM UTC

   CXOne MPower Expert Service Degradation: Sites unavailable. The CXOne MPower Expert Engineering is looking into issues related to slow load times on sites.

2. monitoring Mar 13, 2025, 07:47 PM UTC

   A fix will be implemented while we monitor results.

3. monitoring Mar 13, 2025, 08:06 PM UTC

   A fix will be implemented while we monitor results.

4. monitoring Mar 13, 2025, 08:22 PM UTC

   A fix will be implemented while we monitor results.

5. monitoring Mar 13, 2025, 08:37 PM UTC

   A fix will be implemented while we monitor results.

6. resolved Mar 13, 2025, 08:41 PM UTC

   This incident has been resolved.

7. postmortem Mar 19, 2025, 02:41 PM UTC

   **Impact Start Time \(UTC\) - 03/13/2025 07:42 PM \(UTC\)** **Impact End Time \(UTC\) - 03/13/2025 08:41 PM \(UTC\)** ‌ **Incident Summary** On 3/13/2025, some CXone Mpower customers reported being unable to access the CXone Mpower Expert knowledge portal. The issue stemmed from another routine security patch update implemented by our cloud service provider \(CSP\), following their first update that triggered the first major incident \(02521994. The impact was resolved by refreshing the routing table until the rolling restart was completed, which restored all nodes to normal operation and brought the service back online. ‌ **Root Cause** The issue stemmed from another routine security patch update implemented by our cloud service provider \(CSP\), following their first update that triggered the first major incident \(02521994\). This update triggered the expected rolling restart on all nodes during our internal planned change deployment. While a normal rolling restart typically doesn’t cause impact, some proxy nodes unexpectedly crashed while engineers were refreshing the routing table to prevent disruptions. This resulted in sporadic service disruptions, and engineers had to repeatedly refresh the routing tables until the system fully recovered. Despite this, the Engineering team identified key areas for improvement in the health check mechanisms to prevent similar incidents in the future. ‌ **Corrective Actions** **Detection:** * Internal support teams detected a potentially customer-impacting issue through proactive alarms and monitoring mechanisms, which was later confirmed by customer reports indicating that they were unable to access the CXone Mpower Expert knowledge portal. **Remediation:** * The impact was resolved by refreshing the routing table until the rolling restart was completed, which restored all nodes to normal operation and brought the service back online. Completed on 03/13/2025. **Prevention:** * Following the service restoration, the Engineering team updated the server instance configuration to prevent automatic security patch updates from occurring during release operations. The security update automation will be conducted during scheduled maintenance windows with close supervision to prevent unexpected disruptions in the future. Completed on 03/13/2025. * Engineering team will implement additional health check mechanisms in the application load balancer to verify if the internal load balancers are functioning properly. This enhancement will improve the built-in automatic failover mechanism, allowing the system to switch to a healthy load balancer and prevent customer impact. An update will be provided by End of Day \(EOD\) MT on 03/28/2025. * Engineering team will enhance the internal load balancer updater service to optimize its functionalities and prevent unwanted failures in the future. This will ensure a seamless update during the rolling restart of the nodes. An update will be provided by EOD MT on 05/02/2025. **Risk of Reoccurrence of Impact - Low** ‌ **Incident Timeline \(UTC\)** 03/13/2025 07:42 PM \(UTC\) - Engineers proactively identified a customer-impacting incident via their internal monitoring and began remediation steps. 03/13/2025 07:43 PM \(UTC\) - The first customer case opened, which was later confirmed to be related to this major incident. 03/13/2025 07:47 PM \(UTC\) - While engineers had already identified a suspected cause and were actively working on remediating the impact, they notified the Network Operations Center \(NOC\) engineers about the potentially customer-impacting incident. A proactive major incident was raised and confirmed. 03/13/2025 08:41 PM \(UTC\) - The impact was resolved after the rolling restart was completed, and the routing table was refreshed to restore the affected nodes. Following successful internal validations, the major incident was marked as resolved.

Read the full incident report →

Started Mar 04, 2025, 05:35 PM UTC· resolved Mar 05, 2025, 01:08 AM UTC

1. monitoring Mar 04, 2025, 05:35 PM UTC

   All Services Running Normally

2. monitoring Mar 04, 2025, 05:51 PM UTC

   All Services Running Normally

3. monitoring Mar 04, 2025, 07:14 PM UTC

   All Services Running Normally

4. monitoring Mar 04, 2025, 08:22 PM UTC

   All Services Running Normally

5. monitoring Mar 04, 2025, 09:12 PM UTC

   All Services Running Normally

6. monitoring Mar 04, 2025, 09:28 PM UTC

   All Services Running Normally

7. monitoring Mar 04, 2025, 09:46 PM UTC

   All Services Running Normally

8. monitoring Mar 04, 2025, 10:05 PM UTC

   All Services Running Normally

9. monitoring Mar 04, 2025, 10:20 PM UTC

   All Services Running Normally

10. monitoring Mar 04, 2025, 10:36 PM UTC

    All Services Running Normally

11. monitoring Mar 04, 2025, 10:51 PM UTC

    All Services Running Normally

12. monitoring Mar 04, 2025, 11:06 PM UTC

    All Services Running Normally

13. monitoring Mar 04, 2025, 11:24 PM UTC

    All Services Running Normally

14. monitoring Mar 04, 2025, 11:39 PM UTC

    All Services Running Normally

15. monitoring Mar 04, 2025, 11:55 PM UTC

    All Services Running Normally

16. monitoring Mar 05, 2025, 12:12 AM UTC

    All Services Running Normally

17. monitoring Mar 05, 2025, 12:26 AM UTC

    All Services Running Normally

18. monitoring Mar 05, 2025, 12:42 AM UTC

    All Services Running Normally

19. monitoring Mar 05, 2025, 12:57 AM UTC

    All Services Running Normally

20. resolved Mar 05, 2025, 01:08 AM UTC

    Service continuing without incident. Running notifications are now suspended

Read the full incident report →

Started Mar 04, 2025, 04:02 PM UTC· resolved Mar 04, 2025, 05:34 PM UTC

1. investigating Mar 04, 2025, 04:02 PM UTC

   MindTouch Service Degradation: Sites unavailable. The MindTouch Engineering team is investigating reports of site unavailability.

2. monitoring Mar 04, 2025, 04:30 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Mar 04, 2025, 04:49 PM UTC

   All services running normally

4. monitoring Mar 04, 2025, 05:05 PM UTC

   All services running normally

5. monitoring Mar 04, 2025, 05:20 PM UTC

   All services running normally

6. resolved Mar 04, 2025, 05:34 PM UTC

   This incident has been resolved.

7. postmortem Mar 14, 2025, 03:39 PM UTC

   **Impact Start Time** \(UTC\) 03/04/2025 03:40 PM UTC **Impact End Time** \(UTC\) 003/04/2025 04:05 PM UTC ### Incident Summary On 3/4/2025, some CXone Mpower customers reported inability to access CXone Mpower Expert application sites, where users received error messages intermittently when attempting to login or access the affected sites. Internal teams observed potentially customer impacting issues via proactive monitoring systems, which was later confirmed through customer reports. The impact stemmed from an application capacity issue, which led to the impaired processing capabilities of the application. The impact was resolved when engineers increased the number of application nodes and restarted the application load balancer instances. ### Root Cause The root cause stemmed from an application capacity issue, which was caused by a failover activity performed by our Cloud Service Provider \(CSP\). This activity moved application instances to another host; however, some of the nodes did not return to a healthy state. This caused the load balancer instances to become unstable since it did not have enough instances to process incoming requests. ## Corrective Actions ### Detection: * Internal teams observed potentially customer impacting issues via proactive monitoring systems, which was later confirmed through customer reports of accessibility issues of the CXone Mpower Expert application.. ### Remediation: * Engineers increased the number of application nodes and restarted the application load balancer instances. Completed on 03/04/2025. ### Prevention: * The Engineering team created additional Auto-Scaling Groups \(ASGs\) to segregate workload and prevent negative impact to other workloads when similar issues occur. Completed on 03/14/2025. * Engineers are developing a custom health check, which will be used by the network load balancer to monitor and detect the status of the application load balancers. An update will be provided by End-of-Day \(EOD\) MT of 03/28/2025. ### Risk of Reoccurrence of Impact: Low ### Incident Timeline \(UTC\) 03/04/2025 03:52 PM \(UTC\) - First customer case opened, and Tech Support \(TS\) engineers began the troubleshooting investigation 03/04/2025 04:07 PM \(UTC\) - TS engineers notified the Network Operations Center \(NOC\) engineers about the reported customer impact; a major incident was proposed and confirmed 03/04/2025 04:24 PM \(UTC\) - Engineers identified a suspected cause and began remediation steps 03/04/2025 04:33 PM \(UTC\) - Impact was resolved after engineers identified the issue and implemented a fix and internal tests were successful. Impact and major incident resolved

Read the full incident report →

Started Feb 25, 2025, 11:26 PM UTC· resolved Feb 26, 2025, 07:49 AM UTC

1. investigating Feb 25, 2025, 11:26 PM UTC

   MindTouch Service Degradation: Sites unavailable. The MindTouch Engineering team is investigating reports of site unavailability.

2. identified Feb 26, 2025, 12:26 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Feb 26, 2025, 07:49 AM UTC

   This incident has been resolved.

4. postmortem Mar 04, 2025, 12:16 AM UTC

   ### Impact Start Time \(UTC\) 02/25/2025 11:23 PM UTC Impact End Time \(UTC\) 02/26/2025 07:48 AM UTC ### Incident Summary On 02/25/2025, some CXone Mpower customers reported being unable to access the CXone Mpower Expert knowledge portal, encountering '503 Service Unavailable' and '504 Gateway Timeout' error messages. The impact was caused by the application service cluster, which is responsible for managing cluster infrastructure, entering a failed state after being inadvertently terminated during a routine remediation procedure. The issue was resolved by rebuilding the application service cluster using the latest infrastructure configurations and re-provisioning all necessary resources, including worker nodes and networking settings. ### Root Cause The impact was caused by the application service cluster, which is responsible for managing cluster infrastructure, entering a failed state after being inadvertently terminated during a routine remediation procedure. This led to a service disruption in the US regional platform, causing the applications running on the affected cluster to be unavailable. Although the issue was promptly detected and identified, the Engineering team recognized the need for system and procedural enhancements, particularly for critical processes, and implemented additional safeguards to prevent similar incidents in the future. ## Corrective Actions ### Detection * Internal support teams detected a potentially customer-impacting issue through proactive alarms and monitoring mechanisms, which was later confirmed by customer reports indicating that they were encountering a '503 Service Unavailable' error message when attempting to access the CXone Mpower Expert knowledge portal. ### Remediation * The issue was resolved by rebuilding the application service cluster using the latest infrastructure configurations and re-provisioning all necessary resources, including worker nodes and networking settings. Completed on 02/25/2025. ### Prevention * Based on the lessons learned from this incident, the Engineering team revised the standard operating procedures for executing critical system processes. An additional role was deployed in each production environment to manage and restrict access to manual modifications and updates. Furthermore, termination protection was implemented as an extra safeguard to prevent unintended actions in case of a tool failure, such as a malfunctioning confirmation prompt. Completed on 02/26/2025. ### Risk of Reoccurrence of Impact Low ### Incident Timeline \(UTC\) 02/25/2025 11:23 PM \(UTC\) - Internal support teams detected and notified the Network Operations Center \(NOC\) engineers of a potential customer-impacting issue. At the same time, the first customer case was reported, which was later confirmed to be related to the identified issue. 02/25/2025 11:29 PM \(UTC\) - A major incident was proposed and confirmed. 02/25/2025 11:41 AM \(UTC\) - Engineers engaged with our cloud service provider \(CSP\) support for assistance in identifying the best approach to implement a solution. 02/26/2025 12:26 AM \(UTC\) - Engineers identified the cause of the issue and began their remediation efforts. This took longer than expected to complete since it was updating a large number of service instances. 02/26/2025 03:32 AM \(UTC\) - Engineers re-ran the configuration management process to ensure all resources were properly retained in the system, preventing any unnecessary issues. 02/26/2025 05:30 AM \(UTC\) - The new application service cluster was fully rebuilt, ensuring that all resources were successfully retained. 02/26/2025 06:17 AM \(UTC\) - Engineers continued the remediation and proceeded to run the installation configuration. 02/26/2025 07:48 AM \(UTC\) - The impact was fully resolved after engineers rebuilt the application service cluster with necessary provisioned resources and required configurations. Following successful test validations, the major incident was marked as resolved.

Read the full incident report →

Started Feb 24, 2025, 03:52 PM UTC· resolved Feb 24, 2025, 05:00 PM UTC

1. resolved Mar 04, 2025, 12:11 AM UTC

   - On 2/24/2025, some CXone Mpower customers reported that the CXone Mpower Expert application sites were unavailable. Internal teams observed a potentially customer impacting issue, which was later confirmed via customer reports. The impact stemmed from from some network nodes that were in an unhealthy state. The impact was resolved when engineers restarted the affected network nodes.

2. postmortem Mar 04, 2025, 12:11 AM UTC

   ### Impact Start Time \(UTC\) 02/24/2025 03:52 PM UTC Impact End Time \(UTC\) 02/24/2025 03:57 PM UTC ### Incident Summary On 2/24/2025, some CXone Mpower customers reported that the CXone Mpower Expert application sites were unavailable. Internal teams observed a potentially customer impacting issue, which was later confirmed via customer reports. The impact stemmed from from some network nodes that were in an unhealthy state. The impact was resolved when engineers restarted the affected network nodes. ### Root Cause The root cause stemmed from an issue with the auto-scaling service of the affected application, which caused the service to intermittently fail during high-load times. This resulted in the application falling behind in processing capabilities and led to the drop in traffic to backend services. ## Corrective Actions ### Detection: * Internal teams observed a potentially customer impacting issue, which was later confirmed via customer reports of the unavailability of application sites. ### Remediation: * The auto-scaling service increased the number application instances to handle the incoming traffic. Completed on 02/24/2025. ### Prevention: * The Engineering team removed the auto scaling policy and statically set the nodes far above peak demand and ensure service continuity. Completed on 02/25/2025. * Engineers increased the minimum number of network nodes to address the increase with incoming traffic. Completed on 02/25/2025. ‌ ### Incident Timeline \(UTC\) 02/24/2025 03:51 PM \(UTC\) - Internal teams received a page regarding a potentially customer impacting issue. 02/24/2025 03:52 PM \(UTC\) - Impact start time determined by engineers. 02/24/2025 03:53 PM \(UTC\) - Engineers restarted the affected network nodes in an attempt to resolve the issue. 02/24/2025 03:57 PM \(UTC\) - Impact end time determined by engineers as the system recovered on its own. 02/24/2025 04:00 PM \(UTC\) - Network Operations Center \(NOC\) engineers was informed about the reported customer impact. A major incident was created and confirmed. 02/24/2025 04:01 PM \(UTC\) - As a preventive measure, engineers recycled the router pods. 02/24/2025 04:04 PM \(UTC\) - First customer case opened was reported to the Tech Support \(TS\) engineers. 02/24/2025 04:09 PM \(UTC\) - Engineers continued monitoring and determined that the system is stable. 02/24/2025 04:30 PM \(UTC\) - Customer feedback confirmed that sites were restored. 02/24/2025 05:00 PM \(UTC\) - Internal teams disbanded the Major Incident \(MI\) bridge

Read the full incident report →

Started Jan 30, 2025, 10:57 PM UTC· resolved Jan 31, 2025, 03:18 AM UTC

1. investigating Jan 30, 2025, 10:57 PM UTC

   MindTouch Service Degradation: Sites unavailable. The MindTouch Engineering team is investigating reports of site unavailability.

2. monitoring Jan 30, 2025, 11:14 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 31, 2025, 03:18 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 04, 2024, 08:50 AM UTC· resolved Dec 04, 2024, 09:25 AM UTC

1. investigating Dec 04, 2024, 08:50 AM UTC

   We are currently investigating this issue.

2. investigating Dec 04, 2024, 08:51 AM UTC

   We are continuing to investigate this issue.

3. monitoring Dec 04, 2024, 09:03 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 04, 2024, 09:25 AM UTC

   This incident has been resolved.

5. postmortem Dec 12, 2024, 11:41 PM UTC

   Impact Start Time \(UTC\) 12/04/2024 08:39 AM UTC Impact End Time \(UTC\) 12/04/2024 09:25 AM UTC ‌**Summary** Updated on 12/12/2024 - On 12/04/2024, some NICE CXone customers reported receiving a "504 Gateway Timeout" error when accessing the CXone Expert knowledge portal. The impact stemmed from a procedural error where the sequence of actions performed during the implementation of a planned network change caused misconfigurations on the recently updated subnets following a network node upgrade. The issue was resolved when engineers disabled the misconfigured subnets and then restarted the network service, fully restoring operations. ## **Root Cause** The impact stemmed from a procedural error where the sequence of actions performed during the implementation of a planned network change caused misconfigurations on the recently updated subnets following a network node upgrade. This resulted in a misconfigured subnet routing table, preventing the new subnets from routing to their intended private network route. During the regular rotation of service node assignments as part of an automated system process, some nodes were assigned to the misconfigured subnets. Engineers were not aware that the service node rolling restart automation had to be disabled while updating the subnets, and this step was not included in the procedural guide. As a result, this caused an unwanted network traffic disruption, resulting in the observed impact. Although the issue was quickly detected and resolved, engineers recognized the lessons learned and identified key opportunities for improvement. Preventive measures were put in place to avoid similar issues in the future. ## **Corrective Actions** **Detection** * Internal support teams identified a potentially customer-impacting issue through proactive alarms and monitoring mechanisms, which was later confirmed by customer reports of a "504 Gateway Timeout" error when accessing the CXone Expert knowledge portal. **Remediation** * The issue was resolved when engineers disabled the misconfigured subnets and then restarted the network service, fully restoring operations. Completed on 12/04/2024. **Prevention** * The engineering team updated their procedural guides based on the lessons learned and ensured that all engineers were informed of the new updates to prevent similar issues during future network changes. Completed on 12/11/2024. ## **Incident Timeline \(UTC\)** 12/04/2024 08:39 AM \(UTC\) - Internal support engineers detected the issue and began validating and isolating the cause. Engineers notified the technical groups about the identified incident. 12/04/2024 08:42 AM \(UTC\) - While engineers were actively restoring the service, the first customer case was opened, and Tech Support \(TS\) engineers began troubleshooting the issue. This was later validated and confirmed to be related to the major incident. 12/04/2024 09:03 AM \(UTC\) - Engineers identified a suspected cause and began remediation steps 12/04/2024 09:25 AM \(UTC\) - Impact was resolved after engineers developed and deployed a fix and internal tests were successful. Impact and major incident resolved 12/04/2024 09:54 PM \(UTC\) - A proactive major incident was raised to document this event due to the potential for customer impact, which was later confirmed through customer-reported incident cases.

Read the full incident report →

Started Oct 29, 2024, 11:54 AM UTC· resolved Oct 29, 2024, 01:35 PM UTC

1. investigating Oct 29, 2024, 11:54 AM UTC

   CXOne Engineering is looking into issues related to slow load on MindTouch sites.

2. monitoring Oct 29, 2024, 01:03 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 29, 2024, 01:35 PM UTC

   This incident has been resolved.

Read the full incident report →