CrystalCommerce incident
Admin Outage - Inventory Failing to Save Changes
CrystalCommerce experienced a critical incident on August 9, 2023 affecting Admin and POS and 1 more component, lasting 2d 7h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 09, 2023, 04:11 PM UTC
The main problem we are seeing is that inventory changes cannot be saved. We recommend either not using the POS, or at least make sure to keep track separately of any items on orders placed through the POS, because line items will fail to save onto the orders. This is also preventing any website orders from being placed successfully, so we recommend disabling checkout until we have the problem resolved. We apologize for the trouble this issue is causing, and thank you for your patience. We'll update as soon as we have more info or an ETA.
- monitoring Aug 09, 2023, 04:28 PM UTC
We fixed the issue, and inventory is saving again. I'll work on getting more details from our engineering team for a further update. Right now, they are still working on diagnosing what caused the issue in the first place. Please contact support to let us know if you do still see any issues.
- monitoring Aug 09, 2023, 04:55 PM UTC
IMPORTANT: I'm still seeing issues with line items not saving properly in the POS. Following up with our developers, but it seems the issue may not be completely resolved yet. I'd still recommend not using the POS or keeping track of the line items separately.
- monitoring Aug 09, 2023, 05:23 PM UTC
We are diving deeper into the issue and working on a permanent fix. We may experience interruptions while working on this.
- identified Aug 09, 2023, 07:33 PM UTC
Inventory changes are still not updating. We are working on a stopgap solution, which will be to temporarily disable the activity logs, as well as a permanent fix.
- identified Aug 09, 2023, 08:12 PM UTC
UPDATE: Orders can be placed successfully, both through the POS and frontend site. We are also seeing orders from 3rd party sites come in (eBay, TCGPlayer, etc). They will update inventory properly, but won't show in the activity logs currently. Activity logs are still down, and manual inventory adjustments still cannot be saved. Our plan is to update the activity logs with the order history once we have the underlying issue fixed.
- identified Aug 10, 2023, 07:28 AM UTC
We deployed an update that will allow changes made on the inventory page (e.g. price & qty updates) to save. However, these changes will not show up in the activity logs. This is to restore functionality while we continue to work on the long-term fix. The long-term fix is an upgrade to our database. We anticipate that will be completed tomorrow (Thursday, Aug 10), at which point activity logs will save correctly. Changes from orders placed from now until the upgrade is complete appear on the new logs, but price & qty changes through the inventory page during this window will not be in the logs.
- resolved Aug 11, 2023, 11:19 PM UTC
We have completed the upgrade to our Product Logs database, and product logs are now functioning correctly. Changes from orders placed from the start of the issue now appear on the new logs, but price & qty changes through the inventory page during the window will not be included in the logs.