CrystalCommerce incident
Admin Panel Down - 502/503 Bad Gateway
CrystalCommerce experienced a critical incident on December 8, 2021 affecting Admin and POS and 1 more component, lasting 22h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 08, 2021, 06:00 PM UTC
Our developers are currently investigating the 502/503 bad gateway errors that are preventing anyone from logging into the admin panel.
- investigating Dec 08, 2021, 09:29 PM UTC
We are continuing to work on resolving this issue. While the admin panel is accessible, its performance is degraded. Certain actions, like batch updates, are backlogged. We are also seeing various issues. In particular, the endicia shipping labels are not printing properly. Websites and POS have gone down as well. We'll get them back up as soon as we can.
- identified Dec 09, 2021, 12:40 AM UTC
We've been working on creating a solution based on information from AWS support, and are deploying and upgrade.
- monitoring Dec 09, 2021, 12:57 AM UTC
We found the root cause and deployed a fix. Systems are back online, and we are monitoring.
- resolved Dec 09, 2021, 04:19 PM UTC
This incident has been resolved.