Voyado incident
[Engage] Report Register status and Automation statistics are not working
Voyado experienced a minor incident on March 11, 2025 affecting Other, lasting 3h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 11, 2025, 09:18 AM UTC
We have identified issues with our reports Register status and Automation statistics. They are currently not loading any data. We are working on a fix for this.
- identified Mar 11, 2025, 10:43 AM UTC
Register status is now working. We are still working on a fix for the Automation statistics report.
- resolved Mar 11, 2025, 12:25 PM UTC
Both the Automation statistics report and Register status is now working again.
- postmortem Mar 20, 2025, 02:29 PM UTC
**Summary** On March 11th, two different issues were identified that impacted report functionality within the Engage platform. The first issue affected the Register Status Report by preventing desired report results and the second issue prevented data publication for most tenants in the Automation Statistics Report. **Customer Impact** Customers experienced difficulties accessing accurate reporting data for about a \(1\) day before it was functioning as expected the following day. The respective report was impacted: * Register Status Report * Automation Statistics Report **Root Cause** * **Register Status Report Issue:** A change in how a data type were handled during refactoring led to an oversight in compatibility with an API regarding null values. * **Automation Statistics Report Issue:** Missing necessary tables to be utilized for the report in the production environment data layer. A scheduled job to create these tables was not executed, leading to a mismatch between Engage logic and the Data layer API. **Remediation** Immediate actions were taken to resolve the issues: * **Register Status Report:** A hotfix was deployed to fix the bug. * **Automation Statistics Report:** The necessary job was manually triggered to create the missing tables, restoring the report functionality. **Next Steps** To prevent similar incidents in the future, we have identified improvements to be implemented in areas of testing and validation and to continue to follow established processes. We appreciate your patience and remain committed to delivering a seamless and high-quality platform experience. If you have any further questions or concerns, please reach out to our support team.