CrystalCommerce experienced a minor incident on March 27, 2022 affecting Admin, lasting 16d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 27, 2022, 05:46 PM UTC
Mailgun, our automated email service, temporarily disabled our domain. Their automated system kicked in to prevent it from getting blacklisted. Nothing on our end indicates a change that would have caused this. Mailgun opens at 6AM Monday, and we'll be contacting them right away to get this resolved.
- investigating Mar 28, 2022, 06:12 PM UTC
We were able to get Mailgun to re-enable our account, so automated emails are getting sent again. We are working on resolving the underlying issue. The main reason for the account being shut down was a high bounce rate. So far, the main culprit seems to be "welcome" emails getting sent to invalid addresses.
- monitoring Mar 29, 2022, 09:58 PM UTC
We are running a script to re-send missing order notification emails from when our Mailgun account was disabled. We've lowered our bounce-rate by disabling the customer "welcome" emails. We had an influx of customer accounts created under a certain domain. We'll be blocking that domain to prevent it from happening in the future. If you haven't already, please enable recaptcha v3 for your website. We have a guide for that here: https://help.crystalcommerce.com/en/articles/1868175-register-your-site-to-use-recaptcha-tool
- resolved Apr 12, 2022, 06:13 PM UTC
This incident has been resolved.