Settlement Entries Database Update Impacting Molecule Availability

Update timeline

1. identified Sep 21, 2025, 03:58 PM UTC

   We are currently experiencing an outage with the U.S. application, due to an issue with the database. We have identified the issue and are in the process of deploying a fix. Due to the nature of the issue and the necessary fix, we estimate that the outage may last approximately 8 hours.

2. identified Sep 21, 2025, 05:33 PM UTC

   We’ve implemented a fix for this issue which is related to a change to the underlying Molecule database structure. While the root issue has been identified and a fix is in place, we are experiencing severely degraded performance while the database fix is applied. This process could take up to six more hours to complete. We are monitoring progress, and will provide regular updates on the progress of the fix. Thank you for your continued patience

3. identified Sep 22, 2025, 12:57 AM UTC

   We are still working to restore service, and expect approximately 3-5 hours of additional downtime. The issue is related to running out of available IDs on the primary Valuations table. We will be migrating 30 days of data at a time to the new table, to restore service as quickly as possible. This may result in a delay accessing older data. We will keep you posted as we work through this.

4. resolved Sep 22, 2025, 02:51 AM UTC

   The issue has been fully resolved. The database update and reprocessing have successfully completed in production, and all services are now available. We have verified system stability and data integrity. No further impact is expected. Thank you for your patience while we worked through this incident. We will provide a full post-mortem of the incident in the coming days.

5. postmortem Sep 23, 2025, 02:52 PM UTC

   # **Summary of Incident** On Sunday, September 21, 2025, our system experienced degraded performance impacting valuations in the US production environment. Customers were unable to load the web application or access settlement data. The root cause was identified, a fix applied, and the system was fully restored by 10:45 PM CT. The total duration of the incident was approximately 11 hours. We apologize for the disruption and want to provide a clear summary of what occurred and the steps we’ve taken in response. # **What happened?** A backend service responsible for managing valuations became unavailable due to a database schema change. During the incident, customers were unable to access valuations. Once the migration process completed, service was restored and stability confirmed. # **Why?** The issue was caused by the `id` column in the table reaching the upper limit of its data type \(`INT`\). To accommodate growth, it was migrated to `BIGINT`, requiring a full table rewrite \(~1.2TB\) that locked the system until it completed. # **Corrective/Preventive Actions** To address this issue and prevent similar occurrences, we are: * Reauditing all database tables to proactively migrate any remaining INT sequences. * Migrating settlement entry IDs from INT to BIGINT to ensure long-term capacity. * Implementing monitoring and alerts for sequence usage thresholds. * Updating schema design guidelines so high-ingestion tables default to scalable ID types \(BIGINT\). * Enhancing deployment reviews to ensure large schema changes are rehearsed and applied with minimal disruption. We sincerely apologize for the disruption. While the issue has been resolved with no data loss, we are committed to ensuring this does not recur and are strengthening our systems and processes accordingly