Changes to Trades in the Molecule UI not carrying through to Valuation

Update timeline

1. investigating Oct 21, 2024, 05:20 PM UTC

   We've identified an issue where changes made through the Molecule UI are recognized, and appear on the trade. However, these changes are not carried forward into our valuations table. So the report extracts, valuations pop-ups, and anything else that references valuations is incorrect, for trades that have been manually edited. We are working urgently to identify the full scale and duration of the issue, and to implement a fix

2. identified Oct 21, 2024, 11:02 PM UTC

   We have revalued all trades that were edited through the UI today. We are planning to have a code fix in place tomorrow that will prevent this issue from happening to new trades, and tomorrow we plan to identify and fix any trades that were updated between Oct 11 and yesterday We will continue to monitor

3. resolved Oct 22, 2024, 03:22 PM UTC

   We implemented a code fix last night to prevent new trade updates from having this problem. Today we will run a script to fix trades with unrecognized user updates between 10/11 and 10/20. We will reach out on a customer-by-customer basis if we identify any trades that may cause large PnL swings.

4. postmortem Oct 29, 2024, 01:22 AM UTC

   **Summary of Incident** On October 21, 2024, we experienced an issue related to a recent deployment of a significant upgrade to our custom fields functionality. This issue was limited to trades edited via the Molecule user interface by some users. We apologize for any inconvenience this may have caused and appreciate your patience while we improve our product features. **What happened?** The deployment of a major upgrade to our custom fields features inadvertently created a race condition, resulting in trade data inconsistency. The impact was limited to trades after a trade was edited from the Molecule app UI after October 10, 2024. **Why?** The issue was caused by a coding error in a large-scale code merge related to custom field enhancements. Specifically, a transaction block was added to specific code, intended to improve data integrity, but caused asynchronous processes to fail. **Corrective/Preventive Actions** To address this issue and prevent similar occurrences, we are taking the following actions: * A hotfix was applied to prevent further Trade Edit issues, and the impacted trade data was corrected shortly afterward. * Our upcoming V303 release will include a robust solution addressing the race condition. * We will increase the rigor of our review for large-scale changes and continue our regular practice of breaking down extensive updates into incremental changes where feasible. * Additional monitoring and testing will be developed to verify data consistency during asynchronous processes and to simulate scenarios that may lead to race conditions. We appreciate your understanding as we continuously work to improve the stability and reliability of our services.