Bloomreach incident

EU1 Campaign event triggers workers and IMF oplog lag

Minor Resolved View vendor source →

Bloomreach experienced a minor incident on February 5, 2024, lasting 1h 6m. The incident has been resolved; the full update timeline is below.

Started
Feb 05, 2024, 04:03 AM UTC
Resolved
Feb 05, 2024, 05:10 AM UTC
Duration
1h 6m
Detected by Pingoru
Feb 05, 2024, 04:03 AM UTC

Update timeline

  1. investigating Feb 05, 2024, 04:03 AM UTC

    Campaing event triggers crashlooping on memory. The memory usage is huge. It's being overly processed but still useing huge amount of memory The massive property update is also causing IMF oplog lag Tracking lag currently on 50m

  2. identified Feb 05, 2024, 04:37 AM UTC

    The issue has been identified at 4:30 CET. The issue has been caused by a huge property update campaign that was manually triggered by one of the clients. Lag is still being processed. There is no data loss.

  3. monitoring Feb 05, 2024, 04:38 AM UTC

    Internal brokers and replicas were upscaled to help the lag processing. We are currently monitoring the situation.

  4. resolved Feb 05, 2024, 05:10 AM UTC

    Client has been advised not to use a bad design of the campaigns. The lags are now processed and operation back in normal