[Europe] Data Processing Issue

Update timeline

1. identified Sep 15, 2026, 01:47 PM UTC

   We identified a major incident affecting one data processing server in Europe following a configuration change deployed on September 14 at 09:15 UTC. An incorrect configuration introduced during this change caused unexpected data processing behavior, resulting in data corruption for part of the incoming data. The issue was corrected at **06:56 UTC**. New incoming data has since been processed correctly. Recovery of the affected data started at **08:18 UTC**. As part of the recovery process: * All data received since the configuration change was replayed in chronological order to preserve data consistency. * Corrupted data was identified and cleaned up before replay. Recovery is still ongoing and has been completed for the vast majority of affected data. We are currently investigating a limited number of remaining cases where the replay may not have completed successfully. We will provide a root cause analysis and preventive actions once the investigation is complete. We apologize for the disruption and appreciate your patience.

2. monitoring Sep 15, 2026, 07:09 PM UTC

   Our engineers have deployed a fix and confirmed that the issue has been corrected. The issue was corrected at **06:56 UTC**. New incoming data has since been processed correctly. All data received since the configuration change is being replayed in chronological order to preserve data consistency. We will provide a root cause analysis and preventive actions once the investigation is complete. We apologize for the disruption and appreciate your patience.

3. identified Sep 15, 2026, 07:32 PM UTC

   We identified an issue in the recovery process that prevented some data from being replayed correctly. The issue has now been identified and corrected, and the recovery process has resumed. The affected data is currently being replayed in a controlled manner. Based on the current processing rate, we estimate that the recovery will take approximately 6 hours to complete. We continue to closely monitor the recovery and will provide further updates on its progress.

4. resolved Sep 15, 2026, 09:29 PM UTC

   This incident has been resolved, data is being processed as expected. If doubt, please reach out to your support. We will provide a root cause analysis and preventive actions once the investigation is complete. We apologize for the disruption and appreciate your patience.

5. postmortem Sep 30, 2026, 07:58 AM UTC

   # RCA — Data Integration Incident ### Issue On September 14, a configuration change intended to handle specific special characters during data integration caused some item identifiers to be incorrectly modified. The incident affected approximately 9% of our European installed base. No source data was permanently lost, and the original data required for recovery remained available. ## Analysis The configuration was intended to remove a limited set of special Unicode characters from incoming identifiers. However, the configuration expression was interpreted differently from its intended behavior and caused legitimate individual characters to be removed from some identifiers during integration. As a result, some incoming items were processed with modified identifiers. In certain cases, this also created incorrect associations between item identifiers and their references, which could subsequently affect reference-based label matching. The issue was reported by Support on September 15 at 04:30 UTC and identified at approximately 06:40 UTC. The configuration was corrected, and normal processing of new incoming data was restored by 07:09 UTC on September 15. Recovery of data received during the affected period was then initiated using an existing replay mechanism. During verification, we identified a separate issue in this mechanism that prevented the first recovery attempt from restoring the affected data correctly. The replay mechanism was corrected, and a new controlled recovery was initiated using the original source data. ## Incident Resolution and Recovery The incorrect configuration was removed and normal data integration was restored. Affected data was then cleaned and replayed from the original source data while preserving its chronological order. The final recovery started at 18:30 UTC on September 15 and completed at 02:50 UTC on September 16. The primary item data was recovered from the original source data with no permanent source-data loss. Subsequent verification identified secondary effects on some reference-based associations. A dedicated analysis and remediation process was therefore initiated to identify and correct these inconsistencies. The affected label associations identified through this analysis were corrected. Further verification has since identified that a limited number of residual item/reference inconsistencies may remain in some environments. Additional remediation is ongoing to identify and correct these residual cases. The platform has been processing new incoming data normally since the original configuration was corrected. ## Trigger Cause A configuration change intended to ignore specific special characters during data integration resulted in certain valid alphanumeric characters contained in item identifiers being removed during integration. ## Root Cause The configuration used to handle special characters was interpreted differently from its intended behavior, resulting in unintended modification of item identifiers. The configuration change was tested in pre-production, but the validation did not identify its potential impact on valid item identifiers. Post-deployment monitoring also did not detect the abnormal identifier transformations early enough. A separate defect in the existing replay mechanism complicated the recovery by preventing the initial replay attempt from successfully restoring the affected data. The incident also highlighted that our recovery procedures did not sufficiently account for secondary effects caused by incorrect identifiers, particularly reference-based associations and matching state. These effects required additional analysis after the primary data recovery had completed. ## Corrective and Preventive Actions Immediate corrective actions included correcting the configuration, fixing the replay mechanism, implementing additional recovery monitoring, replaying the affected source data, and correcting the reference-based label associations identified during verification. Following the post-incident review, we are: * Strengthening the review and validation process for production configuration changes, including independent post-deployment monitoring. * Adding safeguards to ensure that valid characters in item identifiers cannot be unintentionally modified. * Strengthening recovery procedures with explicit end-to-end validation of item data, references, and resulting associations. * Improving recovery monitoring so that unsuccessful or incomplete replay operations are detected earlier. * Extending post-recovery verification to identify residual inconsistencies that may not be visible during the primary recovery. * Reviewing recovery strategies for data-corruption incidents to ensure that restoration and deterministic replay options are evaluated early in the incident response. ## Residual Impact and Support The main recovery activities have been completed and the platform is operating normally. However, due to the nature and scale of the original data corruption, a small number of residual inconsistencies may still be discovered during ongoing verification or normal operation. We are continuing our analysis and remediation efforts to identify and correct such cases. If customers observe unexpected item associations, incorrect label content, or other behavior that may be related to this incident, we ask them to contact Vusion Support with the affected store and item or label details. These cases will be investigated and addressed as part of the ongoing remediation. We remain committed to resolving any residual effects identified from this incident.