Inventory Not Saving - Preventing Orders from completing

Update timeline

1. investigating Aug 26, 2021, 05:50 PM UTC

   Inventory changes are not currently saving. This appears to be the same issue with the activity logs we ran into last Monday. Any inventory changes will not save. This means that new orders through your website will not go through, as checkout requires the items to be removed from inventory to complete properly. This also means POS carts will not have line items saved on them and will not update inventory. If you are still using the POS, please keep your own record of the items sold until we can get this resolved. ​ ​We are working on getting this fixed ASAP.

2. identified Aug 26, 2021, 06:22 PM UTC

   We've deployed the update to disable the activity logs which will allow inventory to save again. However, all admin panel and websites are currently down and displaying the "we're sorry, but something went wrong" error message. We are working on getting everything back up as quickly as possible. Then, we will work to re-enable the activity logs.

3. identified Aug 26, 2021, 07:40 PM UTC

   Inventory is saving now, but activity logs remain inaccessible (which we will be resolving as soon as we can). We are also still working on resolving the issue with our production servers (why everything went down completely). As a stop-gap, admins are currently running on our staging server, but as you have probably noticed, it is much slower, and POS + websites are still down.

4. identified Aug 26, 2021, 09:34 PM UTC

   We are continuing to work on a fix for this issue.

5. identified Aug 26, 2021, 11:15 PM UTC

   POS and websites are back up. However the inventory is not saving again. Be aware that this means line items will not be added to POS orders properly. Working with our developers to make sure this issue gets resolved as well.

6. identified Aug 26, 2021, 11:58 PM UTC

   We've disabled the activity logs again, so that inventory can save properly now. We will continue to work on bringing the activity logs back online. While you can make inventory changes now (including through the POS), no activity logs for those changes will be saved.

7. identified Aug 27, 2021, 01:01 AM UTC

   While inventory is saving now, we are still seeing an issue with both the website and POS checkouts not going through. We are investigating, and will update you as soon as we have more info.

8. resolved Aug 27, 2021, 06:40 PM UTC

   All systems are back online. The technical reason for the downtime was due to our Mongo database, which handles all logs of products across our network. There was an issue that caused it to run out of space, bringing everything down. They had to increase the space to combat the issue and then bring the system back online via a series of deployments that initially failed but then succeeded. We would like to apologize for the downtime yesterday and any impact this had on your business.

9. postmortem Aug 27, 2021, 06:40 PM UTC

   All systems are back online. The technical reason for the downtime was due to our Mongo database, which handles all logs of products across our network. There was an issue that caused it to run out of space, bringing everything down. We had to increase the space to combat the issue and then bring the system back online via a series of deployments that initially failed but then succeeded. We would like to apologize for the downtime yesterday and any impact this had on your business.