Molecule experienced a minor incident on July 18, 2025 affecting Molecule, lasting 6h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 18, 2025, 08:42 PM UTC
There is an issue where Molecule is not creating new YTD entries. This affects the Valuation YTD extract as well as any custom reports built around this dataset. We have a plan to fix this Friday evening after hours.
- resolved Jul 19, 2025, 01:24 PM UTC
This incident has been resolved and all YTD valuations are currently up to date.
- postmortem Jul 22, 2025, 04:01 PM UTC
## **Summary of Incident** On July 18, 2025, our system monitoring detected an issue in generating Year-to-Date \(YTD\) records. The issue affected the creation of new YTD records, which may have resulted in incomplete information within reports and dashboards during the affected window. The root cause of the issue was identified, fixed, tested, and scheduled for deployment after business hours. The issue was resolved at 9:47 PM CT. We apologize for the disruption and want to provide a clear summary of what occurred and the steps we’ve taken in response. ## **What happened?** A backend service responsible for storing YTD performance data failed to create new records due to an internal system limitation. This caused downstream reporting jobs that rely on current YTD data to error out, resulting in gaps or failures in customer-facing dashboards and reports. ## **Why?** The issue was caused by a database sequence used to auto-increment IDs in the YTD data table reaching its maximum value. The table was originally configured with an INT data type, which has a finite upper limit. Once that limit was reached, the system was unable to generate new record IDs, causing downstream processes to fail. ## **Corrective/Preventive Actions** To address this issue and prevent similar occurrences, we are taking the following actions: * We’ve migrated the sequence from INT to BIGINT, increasing the ID capacity to accommodate long-term growth. * We’ve reprocessed all failed report jobs to ensure data integrity and completeness for impacted customers. * Initiate a full audit of database tables using integer-based sequences to proactively identify other high-growth tables at risk. * Establish sequence usage monitoring to provide early warnings when thresholds are being approached in the future. * Update data modeling guidelines to ensure high-volume tables use scalable ID types by default going forward Thank you for your patience as we continue to improve the resilience of our systems and take proactive steps to prevent similar issues moving forward.