CrystalCommerce experienced a minor incident on January 18, 2025 affecting Admin and POS and 1 more component, lasting 25d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 18, 2025, 11:19 PM UTC
We’ve identified that system brownouts were being caused by intensive database queries. To address this, we’ve implemented a performance solution that has already resulted in a 90% improvement, and will provide long-term stability improvements for the system. We’re actively investigating the root cause and exploring ways to prevent this from happening again in the future. Thank you for your patience and continued support!
- identified Jan 18, 2025, 11:20 PM UTC
The issue has been identified and a fix is being implemented.
- identified Jan 18, 2025, 11:21 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 21, 2025, 06:45 PM UTC
We have been continuing to work on the 503 errors affecting the frontend websites. We will be deploying an update in the next hour or so that we are expecting to help with the site performance. We also have some other avenues we are looking into as well with our data center contacts today to improve things. Will post more as we have more updates.
- monitoring Jan 21, 2025, 10:53 PM UTC
So far we've been seeing a big improvement after the deploy today. We will keep monitoring things to make sure they are stable, but the "brownouts" seem to have ceased. Please let us know if you do continue to see any issues with sites not loading.
- resolved Feb 12, 2025, 07:53 PM UTC
This incident has been resolved.