Molecule experienced a notice incident on July 1, 2024 affecting Molecule, lasting 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 01, 2024, 07:16 PM UTC
Multiple trades have not valued for today. We are in the process of fixing manually and doing a root cause analysis. We will reach out to individual risk managers as we clear each customer, and we will update this incident as we get a comprehensive solution.
- resolved Jul 01, 2024, 08:07 PM UTC
This incident has been resolved. All current-day valuations line up with expectations. Later today we will run a script to fix any missing place-holder valuations for tomorrow's as_of date, and we will continue monitoring forward-dated valuations
- postmortem Jul 05, 2024, 09:35 PM UTC
# **Incident Report: Incomplete Valuation Data - Resolved** **Date of Resolution:** July 1, 2024 ## **Summary** On Monday morning, July 1, 2024, an issue where certain trades were missing valuation entries was identified and addressed the same day. We apologize for any inconvenience this may have caused and appreciate your understanding. ## **What happened?** The Molecule application has a feature that limits how far into the future valuation entries will be created when trades are made. However, there was no mechanism to create entries for skipped as\_of dates when they became relevant. These previously excluded entries became relevant for the first time on July 1, and some customers began reporting the issue on Friday, June 28. ## **Why** * When implementing the as\_of\_date limits for entry building, a corresponding job to build skipped entries before they became relevant was not available to be run in production. * The system did not include sufficient checks to identify these issues ahead of time. ## **Corrective/Preventive Actions** * A script was deployed immediately to generate missing entries, addressing the issue and correcting the problem for impacted customers as soon as possible. This script will run daily to ensure all entries are built until we remove the limit for good. * An internal check has been created to catch missing entries ahead of time. We apologize for any inconvenience caused and appreciate your understanding as we work to improve our systems and prevent similar issues in the future.