Molecule experienced a major incident on December 3, 2022 affecting Molecule, lasting 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 03, 2022, 12:46 AM UTC
We are currently investigating.
- resolved Dec 03, 2022, 01:17 AM UTC
This incident has been resolved.
- postmortem Dec 05, 2022, 09:58 PM UTC
Last Friday, Molecule experienced a handful of 1- to 4-minute service interruptions between 5:55 PM and 6:59 PM Central Time. We apologize for the interruption, and would like to share the findings from our internal investigation. ## What Happened At 5:55 PM, the Molecule app was unavailable for 4 minutes, because our primary database was suffering excessive load. This happened 3 more times \(with up to 4-minute interruptions each\) over the next hour. ## Why? The database load was primarily due to a mis-timed backup job running during a high-load time for Molecule. The backup is designed to run during the US night, but due to a time zone configuration change \(i.e., US Central to UTC\) it was running during the US day. Complicating this was an unusual set of customer activity querying a large amount of old data \(which is more resource-intensive to query\). ## Preventing This Type of Issue We have, of course, fixed the backup time zone issue. Other items we are working on as well: * Addressing the issue of high-load queries for old data * Ensuring that all of our scheduled jobs are set to the same time zone, to prevent confusion Thank you for your patience, and again--we’re sorry for the interruption.