Bloomreach incident
BRC Site Protection & CMS Maintenance Mode Issues
Bloomreach experienced a major incident on August 8, 2024 affecting US East and US Northeast and 1 more component, lasting 1d 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 08, 2024, 11:40 AM UTC
Dear Customer, We are aware of an issue with our Site Protection and CMS Maintenance Mode features [KB article: https://xmdocumentation.bloomreach.com/bloomreach-cloud/reference-documentation/protect-environment.html] causing downtime during blue/green deployments. We recommend delaying any production deployments if either of these features is enabled in your production Green or Blue environment, until we have deployed a patch. Note: We are actively working on the patch and should be ready shortly. If you must do a production deployment, please disable the Site Protection and CMS Maintenance Mode before starting the deployment process, and only re-enable it after the deployment is complete. We apologize for the inconvenience and will update you as soon as we have more information. Best Regards, Bloomreach Team
- identified Aug 09, 2024, 03:45 PM UTC
We are continuing to work on a fix for this issue.
- resolved Aug 09, 2024, 03:50 PM UTC
Dear Customer, Through our investigation we have not been able to reproduce the exact error using the latest version of the BRC (Bloomreach Cloud) platform. We strongly suspect that if either feature was enabled while the stack was running an older BRC (Bloomreach Cloud) version, that the Kubernetes secret created at that point could be at fault. Therefore for all environments currently using either feature, we have regenerated the secrets as a preventative measure. We have run further checks that there are no references to missing secrets. We do not expect further issues with these features during deployments. If you encounter an issue during a production deployment, please submit a high-priority ticket via [email protected]. For any further questions or additional assistance, please contact the support team. Regards, Bloomreach Team