[Fire platfom] Long delays in updating / performing CRM actions during retry

Update timeline

1. investigating Apr 10, 2025, 07:10 PM UTC

   In our Fire platform we have a mechanism that will retry an CRM action if something fails, or if a Lead doesn't exist yet, and so forth. We have heard that there are long delays in executing these updates which include updates to ownership as well as other fields and creation logic. This is a top-priority issue, and our engineers are investigating the cause as we speak. As reported, this only impacts actions that required a "retry", and is only within our Demand Conversion Platform products (Fire).

2. investigating Apr 10, 2025, 07:56 PM UTC

   We have identified a larger scope for the incident, in that all CRM Actions can be impacted by delays or failures. This can include Lead / Contact creations prior to any Event creation. As a side-effect from some of the issues, Event records have been duplicated in some instances as well. This is following an "at least once" method where it will attempt to create the event in the case that there may have been an issue, to ensure at least one event is created. This is not an intended side-effect, and we are investigating how to mitigate this aspect of the issue as well.

3. monitoring Apr 11, 2025, 03:22 PM UTC

   We've deployed a fix that will help improve the CRM action delays / failures. However, we will need to monitor traffic for at least today in order to determine if the fix ultimately did the trick. We are actively monitoring the situation, and the impact of this fix, and will act quickly if further action needs to be taken.

4. resolved Apr 11, 2025, 04:58 PM UTC

   Our tracking of the delays indicates a near zero-latency after our fix was deployed, so we are considering this incident resolved. If you continue to experience any delays in retries or issues where ownership / lead updates are not performed, please reach out to [email protected] for further assistance.