CrystalCommerce Outage History

Our engineers are aware of the issues with the POS and Admin and are working to resolve it as soon as possible.

We're seeing spikes in website response times again today, resulting in "website is under heavy load" errors. We are continuing to look into this. We'll being deploying some aggressive countermeasures, which will disable product landing pages but at least allow websites to load. This is a short-term solution to get the websites to load. We'll be re-enabling the product landing pages once we can block the offending IPs.

We're seeing another spike in website response times today, resulting in "website is under heavy load" errors. Our team is investigating, and working on blocking whatever is hitting our system. Most likely bots, scrapers, or DDOS.

Our engineering team is working on resolving an issue where frontend websites aren't loading and are instead displaying an error page that says "This website is under heavy load (queue full)" Our system is processing user activity that is far above average, and we are investigating any potential malicious or illegitimate activity. We are also as looking into ways to optimize the system to handle the increase in traffic/activity. Getting websites loading properly is our top priority.

We’ve identified that system brownouts were being caused by intensive database queries. To address this, we’ve implemented a performance solution that has already resulted in a 90% improvement, and will provide long-term stability improvements for the system. We’re actively investigating the root cause and exploring ways to prevent this from happening again in the future. Thank you for your patience and continued support!

We are currently working on resolving an issue with Resque Scheduler, which controls all of the background jobs. That means right now all background jobs are down. This includes batch updates, mass create, order status changes, and automated emails (basically anything that runs behind the scenes). Our top priority is getting this working again. We've tried restarting/redeploying the scheduler, but it's still timing out. We are continuing to investigate what is causing the issue, at which point we will fix it asap.

We are currently seeing an issue with line items not saving properly. If you are using the POS, please be aware that this means inventory changes will not be saved for items you add to the POS cart, and the orders will not have line items after completion. This will also prevent customers from placing orders on the website. We are working on getting this resolved asap. I'm not sure what the ETA is, but it is top priority, and I'll send another message as soon as I have an update. We also have reports that orders are not changing status and new products cannot be created. These are likely related to this issue, and we'll work on getting those actions functional as well.