Bloomreach incident

BRC Site Protection & CMS Maintenance Mode Issues

Major Resolved View vendor source →

Bloomreach experienced a major incident on August 13, 2024 affecting US East and US Northeast and 1 more component, lasting 5d 23h. The incident has been resolved; the full update timeline is below.

Started
Aug 13, 2024, 03:41 PM UTC
Resolved
Aug 19, 2024, 03:33 PM UTC
Duration
5d 23h
Detected by Pingoru
Aug 13, 2024, 03:41 PM UTC

Affected components

US EastUS NortheastUS WestUS NorthwestEU West-1EU CentralAsia Pacific Southeast (SG)

Update timeline

  1. investigating Aug 13, 2024, 03:41 PM UTC

    Dear Customer, We are once again observing issues with our Site Protection and CMS Maintenance Mode features [https://xmdocumentation.bloomreach.com/bloomreach-cloud/reference-documentation/protect-environment.html] causing downtime during Blue/Green deployments. We recommend delaying any production deployments if either of these features is enabled in your Production Green or Blue environment, until we have deployed a patch. If you must do a Production deployment, please disable the Site Protection and CMS Maintenance Mode in both the source and target (Blue and Green) environments, before starting the deployment process, and only re-enable it after the entire deployment process is complete. If you have any further questions or need additional assistance, please reach out to support. Apologies for the inconvenience caused. Best regards, Bloomreach Support Team

  2. identified Aug 14, 2024, 01:32 PM UTC

    Dear Customer, We have identified the bug in the "Set Production Environment" logic [https://xmdocumentation.bloomreach.com/bloomreach-cloud/reference-documentation/mark-an-environment-as-production.html], and we are working on a patch. We expect the patch to be rolled out to all customer stacks within two business days. If you must do a Production deployment, please disable the Site Protection and CMS Maintenance Mode in both the source and target (Blue and Green) environments, before starting the deployment process, and only re-enable it/them after the entire deployment process (including production switch) is complete. If you have any further questions or need additional assistance, please reach out to support. Apologies for the inconvenience caused. Best regards, Bloomreach Support Team

  3. identified Aug 19, 2024, 02:11 PM UTC

    We are currently rolling out a patch release to all customer stacks to resolve this issue. You may notice the maintenance banner in your Mission Control. The rollout should be completed within a few hours.

  4. resolved Aug 19, 2024, 03:33 PM UTC

    All customer stacks have been upgraded to the latest BRC version with a patch for this issue. The Site Protection and CMS Maintenance Mode features should no longer create any issues or outages in conjunction with production environment switches and/or Blue/Green deployments. If you notice any further issues in this area, please contact support.