CrystalCommerce incident
Inventory Not Saving - Preventing Orders from completing
CrystalCommerce experienced a critical incident on March 8, 2022 affecting Admin and POS and 1 more component, lasting 6h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 08, 2022, 06:58 PM UTC
We are experiencing an issue where activity logs are not saving. This is preventing inventory updates from happening, and preventing new orders from being created properly. It is also making it so that website orders can't go past "in_checkout" status, since that requires the quantities to be decremented. If you are using the POS, please be aware that any items added to the POS cart will not be saved properly on the order, so we recommend either manually keeping track of the line items added, or refrain from using the POS until we've resolved the issue.
- monitoring Mar 08, 2022, 08:25 PM UTC
Mongo db is back up. Our testing has shown that inventory changes are saving now. We are still monitoring the situation to make sure that the issue is fully resolved.
- monitoring Mar 08, 2022, 10:03 PM UTC
We got things restored, and we have also found ways to prevent this from occurring again. We had a database primary node get into a bad state because of lack of memory, had to reboot and reconfigure everything. Going to be moving things to a managed service so we won't need to actively manage this internally. Anything that can bring everything down like this is a big deal and we're going to treat it as a top priority. We are investigating potentially missing eBay or TCGPlayer orders from this morning.
- resolved Mar 09, 2022, 01:19 AM UTC
This incident has been resolved.