MindTouch experienced a minor incident on August 13, 2024, lasting 1d 3h. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Aug 13, 2024, 05:27 PM UTC
We are currently experiencing issues with the Export/Import process that will inhibit users from running these processes. The issue has been identified and a fix is being worked on at this time.
- identified Aug 13, 2024, 05:30 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Aug 13, 2024, 09:26 PM UTC
A fix for the import/export issues has been deployed and we are monitoring further.
- identified Aug 13, 2024, 11:33 PM UTC
The issue still persists and we are following up further for a fix.
- identified Aug 13, 2024, 11:33 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Aug 14, 2024, 12:40 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Aug 14, 2024, 03:22 PM UTC
We believe the issue has been resolved and will continue to monitor the situation to ensure stability.
- monitoring Aug 14, 2024, 03:31 PM UTC
We are continuing to monitor for any further issues.
- resolved Aug 14, 2024, 08:52 PM UTC
This incident has been resolved.
- postmortem Aug 14, 2024, 09:08 PM UTC
**Root Cause Analysis:** * During our recent release, the Import/Export tool experienced issues where jobs could start but not complete. This was caused by a regression introduced when updating a third-party backend package. **Recovery:** * At approximately 20240813 21:26 UTC, we performed a targeted deploy of a hotfix to revert a third-party package to a previous stable version. Despite this, the Import/Export tool continued to experience issues. We determined that a full deployment was necessary to fully resolve the problem. At approximately 20240814 02:30 UTC, the full deployment completed, and the issue was resolved. **Corrective Actions** 1. CXone Expert has downgraded third-party package to a stable version until a new, reliable version is available. 2. CXone Expert is adding additional quality tests to our testing suite to ensure this area of import/export is covered.