Bloomreach incident

EXP - Downtime of a database

Major Resolved View vendor source →

Bloomreach experienced a major incident on February 12, 2024 affecting EXP, lasting 1h 59m. The incident has been resolved; the full update timeline is below.

Started
Feb 12, 2024, 04:30 PM UTC
Resolved
Feb 12, 2024, 06:29 PM UTC
Duration
1h 59m
Detected by Pingoru
Feb 12, 2024, 04:30 PM UTC

Affected components

EXP

Update timeline

  1. investigating Feb 12, 2024, 05:28 PM UTC

    We are currently investigating the issue. Impacted are Web personalization (unavailable), analytics (unavailable), and delay on campaigns and tracking.

  2. investigating Feb 12, 2024, 05:32 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Feb 12, 2024, 05:55 PM UTC

    Our developer team has restarted the database and we can see affected components starting to recover. Monitoring to make sure the lag is being processed as expected.

  4. resolved Feb 12, 2024, 06:29 PM UTC

    The database was recovered. Tracking and Imports lag was processed and data ingestion is fully operational. Web experience is fully operational. Campaigns lag was processed and campaigns are fully operational.