CrystalCommerce Outage History
CrystalCommerce major outage · 2 long-running issues View live status →Pingoru has recorded 50 incidents for CrystalCommerce: 4 in the last 12 months and 0 in the last 90 days.
Incidents, newest first
Critical May 20, 2026 MySQL Response Time Spikes resulting in downtime 2h 7m Admin, POS, Frontend
Started May 20, 2026, 12:17 AM UTC· resolved May 20, 2026, 02:25 AM UTC
- investigating May 20, 2026, 12:17 AM UTC
Our engineers are aware of a service interruption on all systems currently (admin, POS, and frontend websites) and are working on getting everything back up and running as soon as possible. This downtime is related to spikes in MySQL response times, which we are investigating the cause of. We will provide another update as soon as they have the errors corrected. Thank you.
- identified May 20, 2026, 12:22 AM UTC
Latest update from our team: "mysql is up again waiting for the pods to become healthy before sending traffic"
- identified May 20, 2026, 01:23 AM UTC
We've traced the problem to our datacenter, and are filing a ticket with them.
- identified May 20, 2026, 01:27 AM UTC
We are continuing to work on a fix for this issue.
- resolved May 20, 2026, 02:25 AM UTC
Earlier today, between approximately 4:15PM and 6:50PM Pacific, some members experienced slower response times or temporary issues accessing parts of their Crystal Commerce storefront and admin tools. We want to share a clear update — and the good news first: the issue is fully resolved and your data is safe. What happened: A hardware-level issue with one of the storage volumes provided by our cloud infrastructure provider, DigitalOcean, caused a database server to become unresponsive. The root cause was at the underlying infrastructure layer — not with your data, and not with our application itself. What we did: Our engineering team identified the source quickly and worked side-by-side with DigitalOcean’s infrastructure team to resolve it. DigitalOcean has confirmed the underlying issue is fully cleared on their side, and we have independently verified every layer of our platform is back to a healthy state. Where we are now: - All services are running normally - Your data is intact — nothing was lost - The database has been verified healthy - The platform is performing as expected - This was an infrastructure availability issue — there are no security or data-integrity concerns Moving forward: Even though the root cause originated outside of our application, we’re treating it as an opportunity to strengthen the platform. Over the coming weeks, we’ll be reviewing our failover and redundancy plans so that any similar provider-side issues in the future have even less impact on you. Thank you for your patience and trust today. If you notice anything unusual or have any questions, please reach out to your account team or our support team — we’re here to help. — The Crystal Commerce Team
Major October 31, 2025 POS and Admin 10h 11m Admin, POS, Frontend
Started Oct 31, 2025, 08:10 PM UTC· resolved Nov 01, 2025, 06:21 AM UTC
- investigating Oct 31, 2025, 08:10 PM UTC
Our engineers are aware of the issues with the POS and Admin and are working to resolve it as soon as possible.
- resolved Nov 01, 2025, 06:21 AM UTC
This incident has been resolved.
Critical September 24, 2025 Frontend Website Downtime 5d Frontend
Started Sep 24, 2025, 05:55 PM UTC· resolved Sep 29, 2025, 05:57 PM UTC
- identified Sep 24, 2025, 05:55 PM UTC
Our websites are currently showing error pages and not loading. Our developers are working on resolving this issue. We attempted a frontend redeployment earlier this morning, but that didn't seem to help Currently, we are doing two things: 1. We'll be deploying the update to disable product landing pages as a temporary stop-gap solution to decrease the load. 2 While that is in place, we are also working on a longer term fix which we anticipate will take a couple hours to deploy (rough estimate: around 1PM Pacific time). Once that is deployed, we'll look into restoring the product landing pages.
- identified Sep 24, 2025, 06:39 PM UTC
Product landing pages (the page a customer goes to when clicking on a product) are disabled, and sites are loading again. We are still working on the other fix, so that we can re-enable product landing pages. Products can still be added to the cart via search results.
- resolved Sep 29, 2025, 05:57 PM UTC
We've resolved the issue. Currently, we have a safeguard in place that will require customer login for viewing product pages. We are still working on building other less intrusive safeguards to protect against future attacks.
Critical September 17, 2025 Websites Under Heavy Load 4h 33m Frontend
Started Sep 17, 2025, 06:07 PM UTC· resolved Sep 17, 2025, 10:40 PM UTC
- investigating Sep 17, 2025, 06:07 PM UTC
We're seeing spikes in website response times again today, resulting in "website is under heavy load" errors. We are continuing to look into this. We'll being deploying some aggressive countermeasures, which will disable product landing pages but at least allow websites to load. This is a short-term solution to get the websites to load. We'll be re-enabling the product landing pages once we can block the offending IPs.
- resolved Sep 17, 2025, 10:40 PM UTC
We have been able to block the attack, and we're increasing the frequency of the IP banner trying to catch the offending IPs earlier.
Critical September 16, 2025 "Website is under heavy load" error on frontend sites 1h 22m Frontend
Started Sep 16, 2025, 06:24 PM UTC· resolved Sep 16, 2025, 07:46 PM UTC
- investigating Sep 16, 2025, 06:24 PM UTC
We're seeing another spike in website response times today, resulting in "website is under heavy load" errors. Our team is investigating, and working on blocking whatever is hitting our system. Most likely bots, scrapers, or DDOS.
- identified Sep 16, 2025, 06:57 PM UTC
We investigated, and the traffic it looks like a new DDOS. We are deploying some countermeasures.
- resolved Sep 16, 2025, 07:46 PM UTC
We've deployed a fix, and websites are loading again now.
Critical August 25, 2025 Frontend Website "under heavy load (queue full)" Errors 1d Frontend
Started Aug 25, 2025, 05:38 PM UTC· resolved Aug 26, 2025, 06:20 PM UTC
- investigating Aug 25, 2025, 05:38 PM UTC
Our engineering team is working on resolving an issue where frontend websites aren't loading and are instead displaying an error page that says "This website is under heavy load (queue full)" Our system is processing user activity that is far above average, and we are investigating any potential malicious or illegitimate activity. We are also as looking into ways to optimize the system to handle the increase in traffic/activity. Getting websites loading properly is our top priority.
- monitoring Aug 26, 2025, 04:11 AM UTC
We've implemented some new security measures and have been able to block multiple IP ranges that were responsible for overloading the system. We are seeing sites loading now. We are continuing to monitor the situation and add addresses to the black list as needed.
- resolved Aug 26, 2025, 06:20 PM UTC
This incident has been resolved.
Minor February 12, 2025 Degraded Website Performance and File Exports 76d 22h Admin, Frontend
Started Feb 12, 2025, 07:59 PM UTC· resolved Apr 30, 2025, 06:39 PM UTC
- identified Feb 12, 2025, 07:59 PM UTC
We are seeing spikes in response times on websites that are leading to "Website is under Heavy Load" errors. We are seeing unusual activity hitting the sites that we are working on resolving. Our File Exports are also currently down. This is related to the frontend performance issues. We have been working on a fix, which is currently on staging servers. We will be deploying it as soon as it passes testing.
- identified Feb 12, 2025, 09:29 PM UTC
File Exports are working again.
- resolved Apr 30, 2025, 06:39 PM UTC
This incident has been resolved.
Minor January 18, 2025 System Brownouts 25d 1h Admin, POS, Frontend, Batch Update
Started Jan 18, 2025, 06:00 PM UTC· resolved Feb 12, 2025, 07:53 PM UTC
- investigating Jan 18, 2025, 11:19 PM UTC
We’ve identified that system brownouts were being caused by intensive database queries. To address this, we’ve implemented a performance solution that has already resulted in a 90% improvement, and will provide long-term stability improvements for the system. We’re actively investigating the root cause and exploring ways to prevent this from happening again in the future. Thank you for your patience and continued support!
- identified Jan 18, 2025, 11:20 PM UTC
The issue has been identified and a fix is being implemented.
- identified Jan 18, 2025, 11:21 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 21, 2025, 06:45 PM UTC
We have been continuing to work on the 503 errors affecting the frontend websites. We will be deploying an update in the next hour or so that we are expecting to help with the site performance. We also have some other avenues we are looking into as well with our data center contacts today to improve things. Will post more as we have more updates.
- monitoring Jan 21, 2025, 10:53 PM UTC
So far we've been seeing a big improvement after the deploy today. We will keep monitoring things to make sure they are stable, but the "brownouts" seem to have ceased. Please let us know if you do continue to see any issues with sites not loading.
- resolved Feb 12, 2025, 07:53 PM UTC
This incident has been resolved.
Critical December 11, 2024 Background Workers Down 22h 5m Admin, POS
Started Dec 11, 2024, 06:15 PM UTC· resolved Dec 12, 2024, 04:21 PM UTC
- investigating Dec 11, 2024, 06:15 PM UTC
We are currently working on resolving an issue with Resque Scheduler, which controls all of the background jobs. That means right now all background jobs are down. This includes batch updates, mass create, order status changes, and automated emails (basically anything that runs behind the scenes). Our top priority is getting this working again. We've tried restarting/redeploying the scheduler, but it's still timing out. We are continuing to investigate what is causing the issue, at which point we will fix it asap.
- identified Dec 11, 2024, 06:51 PM UTC
We've found the root cause of the background workers issue this morning, and are implementing a fix. It may take a while for all of the stalled jobs to catch up, as there are a lot of them in the queue. We are working on getting them all to process them as quickly as possible without overloading the system.
- resolved Dec 12, 2024, 04:21 PM UTC
We got the Resque Scheduler back online, and the backed up queues have cleared now.
Critical December 2, 2024 Line Items not saving 10m Admin, POS, Frontend, Batch Update
Started Dec 02, 2024, 06:54 PM UTC· resolved Dec 02, 2024, 07:05 PM UTC
- investigating Dec 02, 2024, 06:54 PM UTC
We are currently seeing an issue with line items not saving properly. If you are using the POS, please be aware that this means inventory changes will not be saved for items you add to the POS cart, and the orders will not have line items after completion. This will also prevent customers from placing orders on the website. We are working on getting this resolved asap. I'm not sure what the ETA is, but it is top priority, and I'll send another message as soon as I have an update. We also have reports that orders are not changing status and new products cannot be created. These are likely related to this issue, and we'll work on getting those actions functional as well.
- resolved Dec 02, 2024, 07:05 PM UTC
We've deployed a fix, and line items are saving again.