Bloomreach incident

Event ingestion and analytics partial degradation on EXP

Major Resolved View vendor source →

Bloomreach experienced a major incident on May 28, 2024 affecting EXP, lasting 2h 56m. The incident has been resolved; the full update timeline is below.

Started
May 28, 2024, 02:30 AM UTC
Resolved
May 28, 2024, 05:27 AM UTC
Duration
2h 56m
Detected by Pingoru
May 28, 2024, 02:30 AM UTC

Affected components

EXP

Update timeline

  1. investigating May 28, 2024, 04:14 AM UTC

    Because of a user error, an usually high amount of events were ingested which overwhelmed our infrastructure. Impact: slower analytics (reports, metrics etc), and a portion of the tracking might be delayed up to two hours.

  2. monitoring May 28, 2024, 04:17 AM UTC

    The problematic tracking was blocked. We estimate it might take up to two hours to process the lag and for the situation to go back to normal.

  3. resolved May 28, 2024, 05:27 AM UTC

    The lag was entirely processed and the situation is back to normal.