Bloomreach Outage History

Bloomreach is up right now

Bloomreach had 33 outages in the last 2 years totaling 186h 27m of downtime — averaging 1.4 incidents per month.

There were 33 Bloomreach outages since May 28, 2024 totaling 186h 27m of downtime. Each is summarised below — incident details, duration, and resolution information.

Source: https://status.bloomreach.com

Major September 18, 2024

Intermittent Availability Interruption - Bloomreach Dashboard

Detected by Pingoru
Sep 18, 2024, 08:09 AM UTC
Resolved
Sep 18, 2024, 12:01 PM UTC
Duration
3h 51m
Affected: Discovery USDiscovery EMEA
Timeline · 5 updates
  1. investigating Sep 18, 2024, 08:09 AM UTC

    Dear Customer, We have detected an issue in the Bloomreach search and merchandising dashboard. Dashboard users are intermittently facing issues loading the search and merchandising dashboard . Our Engineering team is aware and is actively investigating the issue. Kind regards, Bloomreach Support team

  2. investigating Sep 18, 2024, 09:04 AM UTC

    We are continuing to investigate this issue.

  3. investigating Sep 18, 2024, 10:16 AM UTC

    Dear Customer, Our engineering team had identified the issue and a fix has been applied for the same. We will continue to monitor. Best regards, Bloomreach Support Team

  4. monitoring Sep 18, 2024, 10:17 AM UTC

    Dear Customer, Our engineering team had identified the issue and a fix has been applied for the same. We will continue to monitor. Best regards, Bloomreach Support Team

  5. resolved Sep 18, 2024, 12:01 PM UTC

    Dear Customer, We are pleased to inform you the Bloomreach dashboard is now fully operational. Best regards, Bloomreach Support Team

Read the full incident report →

Notice September 3, 2024

Data processing delay

Detected by Pingoru
Sep 03, 2024, 03:36 PM UTC
Resolved
Sep 03, 2024, 05:08 PM UTC
Duration
1h 32m
Affected: US1
Timeline · 2 updates
  1. investigating Sep 03, 2024, 03:36 PM UTC

    There is a delay in the data processing related to data consumption, including imports and tracking.

  2. resolved Sep 03, 2024, 05:08 PM UTC

    This incident has been resolved.

Read the full incident report →

Notice September 3, 2024

Degraded performance on US1

Detected by Pingoru
Sep 03, 2024, 01:17 PM UTC
Resolved
Sep 03, 2024, 02:07 PM UTC
Duration
50m
Affected: US1
Timeline · 4 updates
  1. investigating Sep 03, 2024, 01:17 PM UTC

    We are currently investigating this issue. Affected modules: imports.

  2. investigating Sep 03, 2024, 01:40 PM UTC

    There are currently delays in tracking and importing processes. The issue is being investigated.

  3. monitoring Sep 03, 2024, 01:54 PM UTC

    A fix has been implemented and we are monitoring the results. The current lag on tracking and imports should be resolved within an hour from now.

  4. resolved Sep 03, 2024, 02:07 PM UTC

    The lag was successfully processed. This incident has been resolved.

Read the full incident report →

Critical August 15, 2024

infinario.com DNS issues

Detected by Pingoru
Aug 15, 2024, 10:31 PM UTC
Resolved
Aug 17, 2024, 05:51 AM UTC
Duration
1d 7h
Affected: CIN
Timeline · 6 updates
  1. investigating Aug 15, 2024, 10:31 PM UTC

    A vendor providing DNS records suspended DNS for infinario.com domains due to reasons currently unknown to us. As a result, on CIN (cloud.infinario.com) domain and any private domain hosted on infinario.com, the engagement platform, it's tracking and external APIs will fail due to DNS resolution. Some users might not yet experience the issues as the DNS might be cached for them, but for any user (including end customer tracking via javascript) without cached DNS, the requests will fail. Our infrastructure team is currently in contact with vendor support to determine the cause and restore the DNS

  2. investigating Aug 15, 2024, 11:38 PM UTC

    We are still in contact with the vendor support. We have not reached resolution with them yet

  3. investigating Aug 16, 2024, 12:39 AM UTC

    After the discussion with the vendor support, we have contacted a specialized team that handles the type of issue that is impacting us. Their expected resolution times are not acceptable, and as such we are preparing another domain as a backup. Customer action will be required for the backup solution to work.

  4. investigating Aug 16, 2024, 12:59 AM UTC

    To mitigate the DNS issues for infinario.com domains and subdomains, as a backup, it's possible to use these domains instead. This applies to the app, tracking to api in all SDKs, etc. cloud.infinario.com -> cin.bloomreach.co api.infinario.com -> cin-api.bloomreach.co public.infinario.com -> cin-cdn.bloomreach.co

  5. investigating Aug 16, 2024, 02:36 AM UTC

    We have changed the default cdn domain to cin-cdn.bloomreach.co. Newly sent emails that do not use custom cnd domain will therefore use this domain for click tracking, etc. Links in emails sent before this change will be unreachable unless the DNS for the original domain is restored.

  6. resolved Aug 17, 2024, 05:51 AM UTC

    The vendor has restored the DNS settings. The DNS now correctly resolve and issue is therefore fixed

Read the full incident report →

Major August 13, 2024

BRC Site Protection & CMS Maintenance Mode Issues

Detected by Pingoru
Aug 13, 2024, 03:41 PM UTC
Resolved
Aug 19, 2024, 03:33 PM UTC
Duration
5d 23h
Affected: US EastUS NortheastUS WestUS NorthwestEU West-1EU CentralAsia Pacific Southeast (SG)
Timeline · 4 updates
  1. investigating Aug 13, 2024, 03:41 PM UTC

    Dear Customer, We are once again observing issues with our Site Protection and CMS Maintenance Mode features [https://xmdocumentation.bloomreach.com/bloomreach-cloud/reference-documentation/protect-environment.html] causing downtime during Blue/Green deployments. We recommend delaying any production deployments if either of these features is enabled in your Production Green or Blue environment, until we have deployed a patch. If you must do a Production deployment, please disable the Site Protection and CMS Maintenance Mode in both the source and target (Blue and Green) environments, before starting the deployment process, and only re-enable it after the entire deployment process is complete. If you have any further questions or need additional assistance, please reach out to support. Apologies for the inconvenience caused. Best regards, Bloomreach Support Team

  2. identified Aug 14, 2024, 01:32 PM UTC

    Dear Customer, We have identified the bug in the "Set Production Environment" logic [https://xmdocumentation.bloomreach.com/bloomreach-cloud/reference-documentation/mark-an-environment-as-production.html], and we are working on a patch. We expect the patch to be rolled out to all customer stacks within two business days. If you must do a Production deployment, please disable the Site Protection and CMS Maintenance Mode in both the source and target (Blue and Green) environments, before starting the deployment process, and only re-enable it/them after the entire deployment process (including production switch) is complete. If you have any further questions or need additional assistance, please reach out to support. Apologies for the inconvenience caused. Best regards, Bloomreach Support Team

  3. identified Aug 19, 2024, 02:11 PM UTC

    We are currently rolling out a patch release to all customer stacks to resolve this issue. You may notice the maintenance banner in your Mission Control. The rollout should be completed within a few hours.

  4. resolved Aug 19, 2024, 03:33 PM UTC

    All customer stacks have been upgraded to the latest BRC version with a patch for this issue. The Site Protection and CMS Maintenance Mode features should no longer create any issues or outages in conjunction with production environment switches and/or Blue/Green deployments. If you notice any further issues in this area, please contact support.

Read the full incident report →

Major July 9, 2024

Feed processing Interruption - Bloomreach Search & Merch

Detected by Pingoru
Jul 09, 2024, 11:57 AM UTC
Resolved
Jul 09, 2024, 01:34 PM UTC
Duration
1h 37m
Affected: Discovery USDiscovery EMEA
Timeline · 3 updates
  1. identified Jul 09, 2024, 11:57 AM UTC

    Dear Customer, We have detected issues with feed processing for Bloomreach Discovery. Other parts of the platform were not affected. Our engineering team has identified the issue and we are actively monitoring the queued feed ingestion jobs and will confirm once incident has been resolved. Regards, Bloomreach Support Team

  2. monitoring Jul 09, 2024, 12:26 PM UTC

    Dear Customer, A fix has been implemented and we are monitoring the feed ingestion queue to confirm this incident has been resolved. Kind regards, Bloomreach Support team.

  3. resolved Jul 09, 2024, 01:34 PM UTC

    Dear Customer, We are glad to inform you the feed processing availability has been restored. Please verify the availability of your Bloomreach applications and let us know if you are still experiencing any problems. We apologise for the inconvenience caused. If you have any further questions or need additional assistance, please reach out to support. Best regards, Bloomreach Support team

Read the full incident report →

Minor June 25, 2024

Search Interruption - Bloomreach Discovery

Detected by Pingoru
Jun 25, 2024, 06:15 AM UTC
Resolved
Jun 25, 2024, 06:41 AM UTC
Duration
26m
Affected: Discovery EMEA
Timeline · 3 updates
  1. investigating Jun 25, 2024, 06:15 AM UTC

    Dear Customer, We are currently experiencing a disruption in the availability of the Bloomreach Search. Our engineering team is involved and actively investigating the issue. We will keep you informed and will update you immediately as soon as we have any more information. Kind regards, Bloomreach Support Team.

  2. monitoring Jun 25, 2024, 06:33 AM UTC

    Dear Customer, The issue was identified and a fix has been implemented and we are monitoring to confirm the incident has been resolved. Kind regards, Bloomreach Support Team.

  3. resolved Jun 25, 2024, 06:41 AM UTC

    Dear Customer, We are glad to inform you the search has been restored. Please verify the availability of your Bloomreach applications and let us know if you are still experiencing any problems. We apologise for the inconvenience caused. If you have any further questions or need additional assistance, please reach out to support. Best regards, Bloomreach Support team

Read the full incident report →

Major May 28, 2024

Event ingestion and analytics partial degradation on EXP

Detected by Pingoru
May 28, 2024, 02:30 AM UTC
Resolved
May 28, 2024, 05:27 AM UTC
Duration
2h 56m
Affected: EXP
Timeline · 3 updates
  1. investigating May 28, 2024, 04:14 AM UTC

    Because of a user error, an usually high amount of events were ingested which overwhelmed our infrastructure. Impact: slower analytics (reports, metrics etc), and a portion of the tracking might be delayed up to two hours.

  2. monitoring May 28, 2024, 04:17 AM UTC

    The problematic tracking was blocked. We estimate it might take up to two hours to process the lag and for the situation to go back to normal.

  3. resolved May 28, 2024, 05:27 AM UTC

    The lag was entirely processed and the situation is back to normal.

Read the full incident report →