Delay in segmentation of the condition "has triggered event with property"

Update timeline

1. identified Apr 04, 2022, 06:41 AM UTC

   On Thursday 31st March at 4:27am UTC we began to experience delays in calculating segments that reference the condition "has triggered event with property". This is a result of a performance degradation on the database responsible for segmentation of events. If your newsletter references a segment that queries for the condition "has triggered event with property" it may be delayed whilst we perform maintenance on this database. We will provide further updates as soon as they become available. Note that automated messages (Workflows, Behavioural and Transactional campaigns) remain **unaffected** and are **operating as normal**. Please email [email protected] if you have any questions regarding your account, segments or newsletters.

2. identified Apr 07, 2022, 09:46 AM UTC

   We've identified the root cause of the performance degradation and we are running maintenance operations to resolve the issue. This maintenance operation will run for the remainder of the week. We will provide another update on progress within 24 hours. In the meantime we've been taking mitigating actions to ensure that affected segments and newsletters continue to process, though you may still experience delays. Please email [email protected] if you have any questions regarding your account, segments or newsletters.

3. identified Apr 08, 2022, 07:49 AM UTC

   Maintenance operations to optimise segment calculation speed are continuing to run. Additionally the system maintenance we will complete this weekend will contribute to improvements. Only the condition "has triggered event with property" is affected with all other segment conditions running efficiently. If you utilise this condition for a newsletter you may see some delay, we'll reach out if this is the case and can provide more information regarding your account. We'll follow up on Monday with an update on the status of this issue and will ofcourse follow up with a detailed postmortem once this is resolved. If you have any questions about your segments or newsletters please email [email protected]

4. monitoring Apr 11, 2022, 09:52 AM UTC

   Maintenance operations to optimise segment calculation speed have completed over the weekend and we are monitoring the results.

5. resolved Apr 12, 2022, 07:12 AM UTC

   Segments with the condition "has triggered event with property" are now calculating in the expected time. This means any related newsletters will also be sending as expected. We'll be following up with a detailed post mortem shortly. In the meantime please contact [email protected] if you have any questions related to your account.

6. postmortem Apr 14, 2022, 05:57 AM UTC

   We're consistently working to evolve Vero's underlying infrastructure to make the platform more efficient and secure. At the end of March we made some changes to the core database powering "event property" conditions. As a result of this change a series of maintenance tasks failed between March 26th and March 31st. As a result, at March 31st at 4:27am UTC we began observing delays with newsletter and segment calculations that relied on the event data property condition. We subsequently diagnosed the root cause of the database performance degradation and fixed and reinstated the maintenance operations. As a significant amount of data had accumulated during the intervening period we had to balance running new processing for segment calculations and the maintenance operations which needed to catch up on the data that had not been processed since the incident first occurred. We saw some gradual improvements as the maintenance progressed and, since this process fully completed \(April 11th, 2:35pm UTC\), we have seen segment calculations return to nominal performance. As part of the \(unrelated\) scheduled maintenance operations that also took place over the weekend of April 10th and 11th we were able to make changes to the processing system that will prevent this occurring again. We are adding further monitoring to ensure any future issues would be detected more promptly. If you have any questions, please contact us at [[email protected]](mailto:[email protected]).