MindTouch Service Degradation: Sites unavailable

Update timeline

1. investigating Sep 05, 2024, 06:33 PM UTC

   We are currently investigating this issue.

2. monitoring Sep 05, 2024, 07:08 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 06, 2024, 12:20 AM UTC

   This incident has been resolved.

4. postmortem Sep 17, 2024, 03:54 PM UTC

   ## **Summary** Updated on 09/10/2024 - On 9/5/2024, some NICE CXone customers reported slowness and login issues on the CXone Expert knowledge platform. The impact was caused by a high volume of traffic, which caused service degradation in the platform. The impact was resolved when engineers permanently blocked the same sources of excessive traffic that had been identified in previous major incidents \(02414694 and 02414906\). ## **Root Cause** The impact was caused by a high volume of traffic, which caused service degradation in the platform. Engineers received multiple alerts and observed spikes in errors, site timeouts, a sudden increase in database connections, and transient traffic. Although the traffic headers appeared to be normal requests and not malicious, the volume was excessively high coming from specific sources. Engineers found that the excessive traffic was originating from the same sources identified in previous major incidents \(02414694 and 02414906\). ## **Corrective Actions** **Detection** * Internal support teams detected a potentially customer-impacting issue through proactive alarm and monitoring mechanisms, which was later confirmed through a customer report of slowness and login issues on the CXone Expert knowledge platform. **Remediation** * The impact was resolved when engineers permanently blocked the same sources of excessive traffic that had been identified in previous major incidents \(02414694 and 02414906\). Completed on 09/05/2024. **Prevention** * Engineers are working to implement additional subnets within the infrastructure and further increase the system capacity to manage a higher volume of traffic. An update will be provided by End of Day \(EOD\) MT on 09/27/2024. ## **Customer Impact** Some NICE CXone customers reported slowness and login issues on the CXone Expert knowledge platform. ## **External Timeline** 09/05/2024 06:33 PM \(UTC\) - Engineers received alarms and started validations and troubleshooting.` `09/05/2024 07:08 PM \(UTC\) - Engineers identified the fix and started the remediation action. The process took time due to the need for a thorough investigation and identification of the sources and the validity of the requests from the identified sources.` `09/05/2024 07:42 PM \(UTC\) - Tech Support \(TS\) engineers notified the Network Operations Center \(NOC\) engineers about the reported related customer cases; a major incident was proposed and confirmed.` `09/05/2024 07:50 PM \(UTC\) - The first customer case opened, and TS engineers began the validation.` `09/06/2024 00:20 PM \(UTC\) - The impact was resolved when engineers permanently blocked the same sources of excessive traffic and internal tests were successful. Impact and major incident resolved.