Bloomreach incident

infinario.com DNS issues

Critical Resolved View vendor source →

Bloomreach experienced a critical incident on August 15, 2024 affecting CIN, lasting 1d 7h. The incident has been resolved; the full update timeline is below.

Started
Aug 15, 2024, 10:31 PM UTC
Resolved
Aug 17, 2024, 05:51 AM UTC
Duration
1d 7h
Detected by Pingoru
Aug 15, 2024, 10:31 PM UTC

Affected components

CIN

Update timeline

  1. investigating Aug 15, 2024, 10:31 PM UTC

    A vendor providing DNS records suspended DNS for infinario.com domains due to reasons currently unknown to us. As a result, on CIN (cloud.infinario.com) domain and any private domain hosted on infinario.com, the engagement platform, it's tracking and external APIs will fail due to DNS resolution. Some users might not yet experience the issues as the DNS might be cached for them, but for any user (including end customer tracking via javascript) without cached DNS, the requests will fail. Our infrastructure team is currently in contact with vendor support to determine the cause and restore the DNS

  2. investigating Aug 15, 2024, 11:38 PM UTC

    We are still in contact with the vendor support. We have not reached resolution with them yet

  3. investigating Aug 16, 2024, 12:39 AM UTC

    After the discussion with the vendor support, we have contacted a specialized team that handles the type of issue that is impacting us. Their expected resolution times are not acceptable, and as such we are preparing another domain as a backup. Customer action will be required for the backup solution to work.

  4. investigating Aug 16, 2024, 12:59 AM UTC

    To mitigate the DNS issues for infinario.com domains and subdomains, as a backup, it's possible to use these domains instead. This applies to the app, tracking to api in all SDKs, etc. cloud.infinario.com -> cin.bloomreach.co api.infinario.com -> cin-api.bloomreach.co public.infinario.com -> cin-cdn.bloomreach.co

  5. investigating Aug 16, 2024, 02:36 AM UTC

    We have changed the default cdn domain to cin-cdn.bloomreach.co. Newly sent emails that do not use custom cnd domain will therefore use this domain for click tracking, etc. Links in emails sent before this change will be unreachable unless the DNS for the original domain is restored.

  6. resolved Aug 17, 2024, 05:51 AM UTC

    The vendor has restored the DNS settings. The DNS now correctly resolve and issue is therefore fixed