CrystalCommerce incident
"This website is under heavy load (queue full)" errors
CrystalCommerce experienced a critical incident on April 21, 2022 affecting Admin and POS and 1 more component, lasting 3h 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 21, 2022, 07:53 PM UTC
Users are experiencing "This website is under heavy load (queue full)" errors when attempting to use CrystalCommerce services. Our engineering team is currently investigating the issue.
- identified Apr 21, 2022, 07:55 PM UTC
Our developers made some adjustments to our server instance count, scaling, and health checks, but we are still getting reports of errors when using the Point of Sale and Admin panel. They are continuing to investigate and make changes as needed.
- monitoring Apr 21, 2022, 08:19 PM UTC
Our engineers need to restart services, you may experience interruptions briefly and then we anticipate errors to subside. We will continue to monitor during this process. We apologize for the interruption.
- resolved Apr 21, 2022, 11:01 PM UTC
We deployed a fix, and the system is running smoothly now. We also added some safeguards for automatically handling long-running queries that can bog things down.