Voyado experienced a minor incident on September 30, 2025 affecting Messaging, lasting 4h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 30, 2025, 07:21 AM UTC
We have identified an issue impacting email sendouts. We are actively investigating and will provide updates on this page as soon as more information is available.
- monitoring Sep 30, 2025, 08:02 AM UTC
A hotfix has been deployed and shows promising results, but unfortunately many messages are still delayed and currently being processed.
- monitoring Sep 30, 2025, 08:39 AM UTC
The incident has been resolved and the deployment was successful. Email sendouts are functioning normally again, but a number of delayed messages are still being processed and delivered.
- monitoring Sep 30, 2025, 09:41 AM UTC
We are continuing to process and deliver the emails that were delayed due to the earlier incident.
- resolved Sep 30, 2025, 11:40 AM UTC
This incident has been resolved. We are continuing to process and deliver the final messages that were delayed due to the incident.
- postmortem Oct 13, 2025, 12:59 PM UTC
## Summary On September 30th, Engage experienced a temporary disruption in email send-outs due to an error in newly deployed code. The issue was quickly identified, and a hotfix was deployed. While no messages were lost, some email messages were delayed as the system worked through a backlog. ## Customer Impact Majority of customers were potentially affected. Most experienced delays in automation emails. In a few cases, delays were up to 8 hours, but the majority of impacted messages were delivered within 3–4 hours. No SMS or priority messages were affected. ## Root Cause The disruption was caused by an internal misconfiguration in how the platform determined where to store and retrieve data for email messages. When a new code deployment corrected this configuration, it unintentionally disrupted messages that had already been processed using the old setup. This led to messages being queued and not sent. ## Mitigation After identifying the issue, a hotfix was deployed to prevent further impact. Additional work was done throughout the day to manually resend affected messages and ensure all queued messages were successfully delivered. ## Next Steps To reduce the risk of similar issues in the future, we are: * Implementing automated code checks that will detect and block misconfigurations before deployment. * Improving internal processes for incident handling and maintenance tooling, including procedures to quickly resending impacted messages. We apologize for any inconvenience caused and appreciate your patience while we resolved the issue.