Molecule Outage History

Started Sep 09, 2026, 08:28 AM UTC· resolved Sep 09, 2026, 08:47 AM UTC

1. investigating Sep 09, 2026, 08:28 AM UTC

   We are currently experiencing an interruption to the ICE connection affecting market date/trade imports across ICE-connected accounts. Our team is actively investigating. No action is needed on your side. Trades and market data will be restored once the connection is restored. We'll post an update as soon as we know more. Thank you for your understanding.

2. resolved Sep 09, 2026, 08:47 AM UTC

   The ICE connection has been restored and market data/trade imports are running normally again across all ICE-connected accounts. Any trades executed during the interruption will be re-fetched. Thank you for your patience.

Read the full incident report →

Started Sep 07, 2026, 07:08 AM UTC· resolved Sep 07, 2026, 07:59 AM UTC

1. investigating Sep 07, 2026, 07:08 AM UTC

   We are aware of an issue currently preventing users in Europe from accessing the Molecule app. Our team is actively investigating the matter, and we will provide an update as soon as we have more information. Thank you for your patience.

2. investigating Sep 07, 2026, 07:39 AM UTC

   We are aware of an issue currently preventing users in Europe from accessing the Molecule app. Our team is actively investigating the matter, and we will provide an update as soon as we have more information. Thank you for your patience.

3. resolved Sep 07, 2026, 07:59 AM UTC

   Molecule EU is now back up and processing normally. There may be a delay as it works through the queue, but P&Ls should be ready shortly.

4. postmortem Sep 11, 2026, 01:13 PM UTC

   # Incident Report: EU Production Unavailability — 7 September 2026 **Status:** Resolved **Duration:** 2 hours, 15 minutes ## Summary On Monday, 7 September 2026, Molecule's EU production environment became unavailable at 07:41 CEST \(05:41 UTC\) and was fully restored at 10:01 CEST \(08:01 UTC\). The cause was growth in valuation and event data that outpaced provisioned storage on that environment. Service was restored by expanding database capacity and draining queued work. Molecule's US and other production environments were unaffected throughout. No customer data was lost or corrupted. ## Timeline \(CEST, UTC\+2\) | Time | Event | | --- | --- | | 07:41 | The EU environment stops serving requests. | | 09:01 | Incident declared and engineering engaged. | | 09:56 | Database capacity expanded; service restored. | | 10:01 | Incident resolved. Queued processing drains normally. | ## What happened Molecule computed the full valuation and settlement history for a set of long-tenor, high-resolution contracts at load time. The YTD aggregations produced by that computation, which are generated at sub-daily resolution, expanded rapidly, and the resulting data growth consumed the storage remaining on the EU environment. The growth was specific to YTD aggregation at sub-daily resolution rather than to the underlying valuation or settlement work. That aggregation is now disabled for large-volume trades while it is reworked. ## Contributing factors Two factors extended the time to restoration: * **Detection did not automatically escalate to a paged incident.** Monitoring identified the condition correctly, but the step from detection to an acknowledged page was manual rather than automatic. * **Authority to make emergency infrastructure changes was not clearly documented.** The fix was correctly diagnosed early, but the change needed to apply it was not obviously within the responder's authority, which added time to mitigation. ## What we are doing about it **Completed** * YTD aggregation is disabled for large-volume trades while the underlying computation is reworked. * Database capacity expanded, with automatic storage scaling enabled on all production environments. * A second tier of escalation is in place when a page goes unacknowledged. * Authority to make emergency infrastructure changes during an incident is documented and findable. * Large data loads are reviewed by our implementations and engineering teams before they run. **In progress \(next 30 days\)** * Automatic incident creation and paging for all detectable severity-level alerts. * Formalizing that review into a documented pre-flight checkpoint, including staging and sign-off. * Expanded on-call coverage for both support and engineering, covering monitoring as well as escalation. * Enforced capacity limits sizing database provisioning against application throughput, applied automatically rather than case by case. * Consolidation of alerting, incident management, and status communication onto a single platform, with a reduction in routine alert volume. **Product changes under development** * An account-level cutover date, so historical trade data loaded for record-keeping purposes does not trigger valuation before a defined date. * Optimization of YTD aggregations for sub-daily resolution trades, which currently generate unnecessarily high data volume. We have scheduled a 30-day review to confirm completion of the items above. We are very sorry for the delays on 7 September, and hope this description and analysis explains how we will prevent a recurrence. ## Questions Customers with questions about this incident should contact their account team or Molecule support.

Read the full incident report →

Started Aug 20, 2026, 02:56 PM UTC· resolved Aug 20, 2026, 03:15 PM UTC

1. investigating Aug 20, 2026, 02:56 PM UTC

   We are currently investigating an issue with our CME trade adaptor that started around 9:30am CST which is affecting our US based customers. We'll provide more information as soon as we can but are working to re-establish our connection.

2. resolved Aug 20, 2026, 03:15 PM UTC

   We resolved an issue with our CME trade adaptor that started around 9:30am CST which affected our US based customers. We have re-established our connection and we have captured all of trades that may have been impacted during the outage. Please let [email protected] know if you discovery anything amiss.

Read the full incident report →

Started Apr 28, 2026, 09:41 PM UTC· resolved Apr 28, 2026, 10:41 PM UTC

1. monitoring Apr 28, 2026, 09:41 PM UTC

   We continue to monitor the Auth0 status but have not registered any impacts to the Molecule platform at this time. However, since user logins may be impacted, we continue to stay alert. Please report any login issues to [email protected] immediately.

Read the full incident report →

Started Apr 15, 2026, 11:07 AM UTC· resolved Apr 15, 2026, 12:58 PM UTC

1. identified Apr 15, 2026, 11:07 AM UTC

   We have identified the issue causing the current outage of our reporting capabilities within Molecule. This is isolated to reports and all other functionality remains functional. We are working on a resolution and will be back with you shortly.

2. identified Apr 15, 2026, 11:33 AM UTC

   We are working to resolve the issue causing the current outage of our reporting capabilities within Molecule. This is isolated to reports and all other functionality remains functional. We are working on a resolution and will be back as soon as we have an update.

3. identified Apr 15, 2026, 12:05 PM UTC

   Molecule's reporting functionality is currently unavailable. We are actively working with our third-party vendor to resolve the issue. All other features remain fully operational. We will provide an update as soon as possible.

4. identified Apr 15, 2026, 12:33 PM UTC

   Our team is actively working with our third-party vendor to restore reporting services as quickly as possible. All other functionality remains unaffected. We appreciate your patience and will share another update shortly.

5. resolved Apr 15, 2026, 12:58 PM UTC

   Reporting services have been fully restored. We appreciate your patience while we worked to resolve this issue and apologize for any disruption to your workflow.

Read the full incident report →

Started Feb 13, 2026, 09:25 AM UTC· resolved Feb 13, 2026, 10:16 AM UTC

1. investigating Feb 13, 2026, 09:25 AM UTC

   We are currently experiencing a technical issue which has prevented the Morningstar settlement market data from updating for yesterday (2026-02-12). Separately, no copied market data has been generated for today. Our Engineering team is actively investigating the issue. We will provide an update as soon as the matter has been resolved. Thank you for your understanding.

2. investigating Feb 13, 2026, 09:50 AM UTC

   Market data is loading on delay. Performance is degraded due to a large market file. We have scaled up our processing workings to handle the load. We investigating further to ensure full processing of market files and will continue to monitor until we are confident this is resolved. Everything else is up, working and accessible.

3. monitoring Feb 13, 2026, 10:02 AM UTC

   We are monitoring the market data which continues to load after we scaled up our workers to handle the processing. A large data file backed up our processing of market data which we are now steadily processing. We are actively checking data across all customers to ensure market data has loaded. Please contact [email protected] if you are missing market data.

4. resolved Feb 13, 2026, 10:16 AM UTC

   An EU Severity 1 incident occurred where market data is not loading or copying, causing downtime for customers; the issue was triggered by a web alert and began at 9:22:03 AM UTC on 2/13/2026 (the incident was created by a web alert at 8:51:02 AM UTC); the cause is EU workers not scaling up because the Horizontal Pod Autoscaler didn't obtain CPU metrics, and the response involved manually scaling the workers to process the market data queue first and verifying customer accounts. Further root cause analysis will be performed and actions taken to ensure this scaling issue doesn't reoccur. We apologize for any interruption in your daily business.

Read the full incident report →

Started Jan 27, 2026, 02:51 PM UTC· resolved Jan 27, 2026, 03:46 PM UTC

1. investigating Jan 27, 2026, 02:51 PM UTC

   We are aware of an issue impacting trade processing times on the Molecule platform. Trades are being received and captured as expected, however some jobs—such as leg building—are experiencing extended queue times. The team is actively investigating and working to restore normal processing performance. Further updates will be shared as progress is made.

2. investigating Jan 27, 2026, 03:17 PM UTC

   The underlying issue causing trade processing delays has been identified, and corrective actions are currently underway. Trades will begin processing as the system recovers, and remaining queued work will be completed. We will share a final update once the issue is fully resolved and normal performance is restored.

3. resolved Jan 27, 2026, 03:46 PM UTC

   Trade processing has returned to normal and all queued jobs are now being processed as expected. We are continuing to closely monitor the platform to ensure stability and prevent a recurrence. Thank you for your patience.

4. postmortem Jan 29, 2026, 03:28 PM UTC

   # **Summary** On January 27, 2026, some customers experienced missing or delayed trade valuations due to a performance issue in our market-data processing pipeline. ## **What Happened?** A recent database upgrade exposed an inefficiency in our market-data copier, leading to excessive memory usage for certain large \("big-bang"\) accounts. This slowed downstream systems, causing valuation jobs to stall or fail. The incident was mitigated by 9:35 AM CT the same day. ## **Root Cause** The issue stemmed from a systemic regression in the market-data copier, where a validation step loaded large datasets into memory. A change in the database query planner post-upgrade exacerbated this behavior, overwhelming system resources and delaying processing pipelines. ## **Impact** * Some trades did not generate legs or valuation results. * Affected customers saw blank or missing valuations during the incident window. ## **Immediate Mitigations** * Declared a SEV-0 and initiated incident response. * Scaled workers and reduced data ingress to stabilize the system. * Cleared backlogs and reprocessed affected trades. ## **Permanent Fixes & Preventive Actions** * Optimized memory-intensive validation logic with a more efficient approach. * Enhanced retry logic and queue handling for workers and RabbitMQ. * Added a set of new early-warning alerts that trigger on conditions that may indicate a problem. * Published internal response playbooks and added auto-scaling for queue backlogs. * Introduced new load tests and validation specifically for large account scenarios. * Reprocessed impacted trades and validated key fields. Support will confirm with affected customers directly. We apologize for the disruption and remain committed to strengthening the resilience of our platform. For any trade-specific questions, please reach out to Support.

Read the full incident report →

Started Oct 20, 2025, 01:23 PM UTC· resolved Oct 21, 2025, 12:10 AM UTC

1. investigating Oct 20, 2025, 01:23 PM UTC

   We are currently experiencing an issue with our cloud provider that is causing delays in data processing. We are working to resolve this as quickly as we can and will continue to post periodic updates as we have them. We appreciate your patience as we work through this.

2. identified Oct 20, 2025, 05:39 PM UTC

   Our cloud provider is continuing to work through a widespread issue impacting the region where our U.S. application runs. We spent the morning testing solutions to run resources in a cloud region that is not impacted and we are confident that we can do so if the need arises. Up to this point, we have not had a major impact to our application, however, workload typically increases in the afternoon when end-of-day data processing begins. We are prepared to deploy additional resources in the second region to keep up with data processing. We will continue to provide updates throughout the afternoon.

3. monitoring Oct 20, 2025, 08:20 PM UTC

   Our cloud provider has addressed the issue and we can confirm improvements. Earlier this afternoon, we deployed additional resources to a second region to keep up with workload. We will continue to monitor our resources closely and provide periodic updates.

4. resolved Oct 21, 2025, 12:10 AM UTC

   This incident has been resolved. We appreciate your patience while working through this.

Read the full incident report →

Started Sep 30, 2025, 10:13 AM UTC· resolved Sep 30, 2025, 11:00 AM UTC

1. investigating Sep 30, 2025, 10:13 AM UTC

   We are currently experiencing ICE adaptor stability issues. This might cause a delay in trades flowing in from ICE. We are currently instigating this as a priority.

2. resolved Sep 30, 2025, 11:00 AM UTC

   The ICE market data feed was down for multiple accounts due to a stalled connection between customer accounts and the ICE FIX listener. The issue was resolved by manually restarting the service for each affected account.

3. postmortem Oct 03, 2025, 10:08 PM UTC

   ## **Summary of Incident** On September 30 and again on October 2, Molecule experienced a brief interruption with the ICE market data feed. Both of these occurances were resolved within an hour or less. ## **What happened?** In both instances, connections to ICE stalled, and real-time pricing and trade ingestion were interrupted. The issue was resolved by re-establishing ICE connections, and any missed trades were successfully reprocessed after the service was restored. ## **Why?** A failure in the ICE FIX listener caused the incident. That is the service that maintains connectivity between Molecule and ICE. The listener encountered a resource exhaustion issue that caused connections to stall without automatically recovering. ## **Resolution** Our team restored service quickly by restarting ICE connectivity for affected accounts. Once connections were re-established, all missed trades were ingested and processed successfully. ## **Next Steps** We are implementing improvements to prevent recurrence: * Infrastructure updates: Increasing system resources and tuning the ICE listener to handle higher load. * Automatic recovery: Adding self-healing logic to re-establish connections without manual intervention. * Monitoring improvements: Enhancing alerts to detect early warning signs and automatically restart services if needed. We sincerely apologize for any disruption. While the issue has been resolved with no data loss, we are committed to ensuring this does not recur and are strengthening our systems and processes accordingly.

Read the full incident report →

Started Sep 21, 2025, 03:58 PM UTC· resolved Sep 22, 2025, 02:51 AM UTC

1. identified Sep 21, 2025, 03:58 PM UTC

   We are currently experiencing an outage with the U.S. application, due to an issue with the database. We have identified the issue and are in the process of deploying a fix. Due to the nature of the issue and the necessary fix, we estimate that the outage may last approximately 8 hours.

2. identified Sep 21, 2025, 05:33 PM UTC

   We’ve implemented a fix for this issue which is related to a change to the underlying Molecule database structure. While the root issue has been identified and a fix is in place, we are experiencing severely degraded performance while the database fix is applied. This process could take up to six more hours to complete. We are monitoring progress, and will provide regular updates on the progress of the fix. Thank you for your continued patience

3. identified Sep 22, 2025, 12:57 AM UTC

   We are still working to restore service, and expect approximately 3-5 hours of additional downtime. The issue is related to running out of available IDs on the primary Valuations table. We will be migrating 30 days of data at a time to the new table, to restore service as quickly as possible. This may result in a delay accessing older data. We will keep you posted as we work through this.

4. resolved Sep 22, 2025, 02:51 AM UTC

   The issue has been fully resolved. The database update and reprocessing have successfully completed in production, and all services are now available. We have verified system stability and data integrity. No further impact is expected. Thank you for your patience while we worked through this incident. We will provide a full post-mortem of the incident in the coming days.

5. postmortem Sep 23, 2025, 02:52 PM UTC

   # **Summary of Incident** On Sunday, September 21, 2025, our system experienced degraded performance impacting valuations in the US production environment. Customers were unable to load the web application or access settlement data. The root cause was identified, a fix applied, and the system was fully restored by 10:45 PM CT. The total duration of the incident was approximately 11 hours. We apologize for the disruption and want to provide a clear summary of what occurred and the steps we’ve taken in response. # **What happened?** A backend service responsible for managing valuations became unavailable due to a database schema change. During the incident, customers were unable to access valuations. Once the migration process completed, service was restored and stability confirmed. # **Why?** The issue was caused by the `id` column in the table reaching the upper limit of its data type \(`INT`\). To accommodate growth, it was migrated to `BIGINT`, requiring a full table rewrite \(~1.2TB\) that locked the system until it completed. # **Corrective/Preventive Actions** To address this issue and prevent similar occurrences, we are: * Reauditing all database tables to proactively migrate any remaining INT sequences. * Migrating settlement entry IDs from INT to BIGINT to ensure long-term capacity. * Implementing monitoring and alerts for sequence usage thresholds. * Updating schema design guidelines so high-ingestion tables default to scalable ID types \(BIGINT\). * Enhancing deployment reviews to ensure large schema changes are rehearsed and applied with minimal disruption. We sincerely apologize for the disruption. While the issue has been resolved with no data loss, we are committed to ensuring this does not recur and are strengthening our systems and processes accordingly

Read the full incident report →

Started Jul 18, 2025, 08:42 PM UTC· resolved Jul 19, 2025, 02:45 AM UTC

1. identified Jul 18, 2025, 08:42 PM UTC

   There is an issue where Molecule is not creating new YTD entries. This affects the Valuation YTD extract as well as any custom reports built around this dataset. We have a plan to fix this Friday evening after hours.

2. resolved Jul 19, 2025, 01:24 PM UTC

   This incident has been resolved and all YTD valuations are currently up to date.

3. postmortem Jul 22, 2025, 04:01 PM UTC

   ## **Summary of Incident** On July 18, 2025, our system monitoring detected an issue in generating Year-to-Date \(YTD\) records. The issue affected the creation of new YTD records, which may have resulted in incomplete information within reports and dashboards during the affected window. The root cause of the issue was identified, fixed, tested, and scheduled for deployment after business hours. The issue was resolved at 9:47 PM CT. We apologize for the disruption and want to provide a clear summary of what occurred and the steps we’ve taken in response. ## **What happened?** A backend service responsible for storing YTD performance data failed to create new records due to an internal system limitation. This caused downstream reporting jobs that rely on current YTD data to error out, resulting in gaps or failures in customer-facing dashboards and reports. ## **Why?** The issue was caused by a database sequence used to auto-increment IDs in the YTD data table reaching its maximum value. The table was originally configured with an INT data type, which has a finite upper limit. Once that limit was reached, the system was unable to generate new record IDs, causing downstream processes to fail. ## **Corrective/Preventive Actions** To address this issue and prevent similar occurrences, we are taking the following actions: * We’ve migrated the sequence from INT to BIGINT, increasing the ID capacity to accommodate long-term growth. * We’ve reprocessed all failed report jobs to ensure data integrity and completeness for impacted customers. * Initiate a full audit of database tables using integer-based sequences to proactively identify other high-growth tables at risk. * Establish sequence usage monitoring to provide early warnings when thresholds are being approached in the future. * Update data modeling guidelines to ensure high-volume tables use scalable ID types by default going forward Thank you for your patience as we continue to improve the resilience of our systems and take proactive steps to prevent similar issues moving forward.

Read the full incident report →

Started Oct 23, 2024, 12:46 PM UTC· resolved Oct 23, 2024, 05:57 PM UTC

1. investigating Oct 23, 2024, 12:46 PM UTC

   We are aware of an issue related to `user` market data updating correctly. We are investigating as a priority.

2. resolved Oct 23, 2024, 05:57 PM UTC

   We have identified the root cause of this issue and have successfully updated and validated the impacted marks. The affected trades have been revalued. We will share details in a post-mortem soon. Thank you for your patience and understanding during this process.

3. postmortem Oct 29, 2024, 01:26 AM UTC

   **Summary of Incident** On October 23, 2024, a system issue impacted the processing of specific background jobs in our European production environment. We apologize for any disruption this may have caused and thank you for your patience. **What happened?** After a hotfix deployment to address a prior incident, the workers responsible for processing job queues in our European production environment failed to resume operations, leading to delays and inaccuracies in market data updates. **Why?** The issue was due to a deployment anomaly. Background job processors did not automatically scale back up after a system update, a step typically handled by our automated scaling process. This error was isolated to the European environment. **Corrective/Preventive Actions** To mitigate this issue and prevent recurrence, we are implementing the following actions: * We are conducting further analysis to determine the precise cause of the workers' scaling error. * Additional monitoring and alerts will be implemented to verify worker scaling after each deployment. * We will review our deployment playbook to ensure confirmation of worker functionality post-deployment and ensure consistency across all environments. We sincerely appreciate your patience as we work to strengthen our systems' resilience and prevent similar disruptions in the future.

Read the full incident report →