[Engage] Issues With Order V3

Update timeline

1. identified May 20, 2025, 07:20 AM UTC

   We are currently investigating an issue with Order V3 where the orders is not processed. No order requests are lost, they are stuck in processing.

2. monitoring May 20, 2025, 09:46 AM UTC

   The issue has been resolved, and order processing has returned to normal. Orders affected during the incident are being queued for handling shortly.

3. resolved May 20, 2025, 11:22 AM UTC

   The incident has now been resolved. Orders affected during the incident have been successfully processed.

4. postmortem May 22, 2025, 10:47 AM UTC

   ### Summary On May 20th 2025, an issue in the Engage platform led to delays in how orders were processed and messages were sent. While everything looked normal on the surface — meaning actions appeared to be accepted — the actual processing behind the scenes wasn’t happening as it should. The issue was fully resolved within a few hours and no orders were lost. ### Customer Impact A small number of customers using specific order features were affected. Orders submitted to Engage were **not processed** during the incident window. As a result, **automated transactional emails** linked to these orders were also delayed. Once the issue was fixed, **all delayed orders were processed successfully** and the related emails were sent. ### Root Cause and Mitigation A recent system changes introduced this problem. Once the issue was identified, our team: * Released a fix that allowed the system to handle orders as expected again. * Made sure all delayed orders were caught up and properly processed.. ### Next Steps We’re adjusting our internal testing setups to more closely match the real-world environment, helping us catch potential issues for this type of case before they can affect you. We appreciate your patience and understanding, and apologize for any inconvenience. We remain committed to providing a stable and reliable platform experience.