Bloomreach experienced a critical incident on November 20, 2023 affecting EXP, lasting 4h 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 20, 2023, 04:54 PM UTC
We are currently experiencing an outage on the EXP instance. The team is working on a fix.
- investigating Nov 20, 2023, 05:34 PM UTC
Teams are still investigating and trying to identify the root cause. The scale of impact and ETA is currently not known. We will provide an update at 19:30 CET
- investigating Nov 20, 2023, 06:31 PM UTC
Our engineering teams continue to investigate to identify the root cause. We anticipate a potential solution within the next 60 minutes. Our next update is scheduled for 20:30 CET, or earlier if there are significant changes in the situation.
- identified Nov 20, 2023, 07:04 PM UTC
Our engineering teams have implemented the fix, and platform recovery is now in progress. We anticipate the platform's functionality to be restored within the next 60 minutes. At the moment, the accommodated lag is not yet determined. We'll communicate this information after the platform is fully back online.
- identified Nov 20, 2023, 08:07 PM UTC
The platform's recovery is slower than anticipated. Although the application is operational, it's experiencing slowness, resulting in a complete halt of campaign processing. Our next update is scheduled for 22:00 CET or earlier if there are significant changes in the situation.
- monitoring Nov 20, 2023, 08:38 PM UTC
We expect the tracking and import lag to be resolved within the next 2 hours. Campaigns are gradually resuming operations and processing accumulated lag, with an estimated completion within the next 2 hours. Regrettably, failed campaigns will require manual restarts. Our next update is scheduled for 23:30 CET or earlier if there are significant changes in the situation.
- resolved Nov 20, 2023, 09:23 PM UTC
The lag has been resolved, and the platform is now fully back online. The root cause analysis will be shared with Customer Success Managers in the upcoming days.