Voyado experienced a minor incident on November 7, 2025 affecting Automations, lasting 4h 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 07, 2025, 11:42 AM UTC
We are currently experiencing a delay in processing events, which impacts features such as Interaction and Target Audience triggers in Automations. Our team is preparing a hotfix to address the issue. We will continue to provide updates as the situation progresses.
- monitoring Nov 07, 2025, 03:00 PM UTC
A fix has been implemented, and we are now seeing improvements in event processing times. Features such as Interaction and Target Audience triggers in Automations are gradually returning to normal performance. We continue to monitor the system closely. The underlying root cause is still under investigation, and we’ll provide more information as it becomes available.
- resolved Nov 07, 2025, 03:54 PM UTC
The issue causing delays in event processing has now been resolved. All systems are operating as expected. Root cause analysis will be included in Post-Mortem.
- postmortem Nov 13, 2025, 08:18 AM UTC
## Summary On November 7th, an unusually high number of internal events were generated due to a planned feature rollout. This caused delays in processing certain automations. ## Customer Impact Automations triggered by customer interactions or audience targeting were delayed by up to 12 hours. All affected automations were eventually executed. Other parts of the platform operated as normal. ## Root Cause and Mitigation When the new Marketing Groups feature was enabled for all customers, a large volume of events \(hundreds of millions\) was generated in a short time. The system processing these events could not keep up, which resulted in delays for other events in the same internal queue. To restore normal flow, we temporarily cleared out the backlog of non-critical events and performed database synchronizations to ensure data accuracy. Normal operation resumed later the same day. ## Next Steps We are improving our monitoring and alerting to detect similar issues earlier, and we are reviewing our event handling capacity and release procedures to prevent future impact from large-scale rollouts. We apologize for any inconvenience caused and appreciate your patience while we resolved the issue.