Nosto incident
We have identified issue causing degraded functionality in the Nosto Commerce Experience Platform - Nosto Search & Category Merchandising
Nosto experienced a critical incident on October 10, 2024 affecting Search & Categories, lasting 4h 13m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 10, 2024, 04:26 PM UTC
We’ve identified what is causing the issue and are working on a to resolve the issue. An update will be provided once the issue has been resolved. Your patience is appreciated. Nosto Search & Category Merchandising. We are investigating the issue.
- identified Oct 10, 2024, 04:32 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Oct 10, 2024, 05:45 PM UTC
We have restored an backup of the Search & Merchandising products and will keep monitoring the issue. We are currently rebuilding the indices with latest product data.
- monitoring Oct 10, 2024, 06:49 PM UTC
It would seem that the site experience has returned to normal state for most merchants we are still proceeding in full re-indexing of products to ensure that latest updates are incorporated to the Search & Merchandising
- resolved Oct 10, 2024, 08:39 PM UTC
We are doing a full product re-index that should fix the remaining discrepancies but for now the Search & Merchandising should be returning correct, but in some cases slightly outdated data.
- postmortem Oct 11, 2024, 03:21 PM UTC
On October 10th, our Search & Category Merchandising service experienced an issue where our automated full indexing process had an problem which caused products in the Search & Merchandising database to be stored in an incomplete state. This led to product data not being displayed for many merchants. After identifying the source of the issue, we restored a snapshot backup data prior to the failed full indexing process. This allowed Search & Merchandising to return to normal operations and serve customer requests, although most recent product updates were not included in the snapshot data backup. We disabled the full indexing until we were sure to have measures in place that prevent similar issues reoccurring. After the preventive measures were implemented, we will re-index all products with the latest data by October 11th \(GMT\). The issue in general was isolated only on customer that were using Search & Category Merchandising. We have a detailed list of actions that we are already implemented to ensure smooth operations and product updates to occur without issues.