Molecule incident
Several app screens are unresponsive. New users unable to log in
Molecule experienced a critical incident on April 29, 2024 affecting Molecule, lasting 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 29, 2024, 04:44 PM UTC
We are investigating with extreme urgency
- investigating Apr 29, 2024, 04:55 PM UTC
App is down. We continue to investigate with extreme urgency
- investigating Apr 29, 2024, 05:15 PM UTC
Underlying issue has been resolved. App is operational. Some data may be stale as processing queues were delayed during the outage. We continue to monitor
- resolved Apr 29, 2024, 05:21 PM UTC
We continue to monitor
- postmortem May 01, 2024, 12:14 AM UTC
**Summary** On April 29, 2024, from 11:34 AM to 12:05 PM CST, a service interruption affected our US-based users. We apologize for any inconvenience. **What happened?** The system was intermittently unavailable to US-based users for approximately 30 minutes. **Why?** The outage was triggered by a large trade upload, which consumed significant system resources. We mitigated the issue by scaling down the heavy resource consumption to allow the system to get back to a healthy state. **Corrective/Preventive Actions** We are committed to providing and continuously improving reliable and efficient software, and this incident has highlighted areas for improvement. To prevent future occurrences of similar issues, we are taking the following steps: 1. **Software Updates**: We are correcting the issue that caused the heavy consumption of resources. 2. **Enhanced System Monitoring:** We are focused on improving our system monitoring to detect and respond to such anomalies before they impact service availability. 3. **Performance Enhancements:** We will roll out several performance improvements over the next few weeks to enhance system resource utilization efficiency. Thank you for your patience.