Voyado experienced a critical incident on April 30, 2025 affecting API, lasting 1h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 30, 2025, 02:09 PM UTC
We are currently investigating a possible incident affecting our APIs.
- identified Apr 30, 2025, 02:20 PM UTC
We have identified the issue and are working on a fix.
- identified Apr 30, 2025, 02:50 PM UTC
The fix is currently being rolled out, and we're starting to see signs of improvement. However, the issue is not fully resolved yet.
- monitoring Apr 30, 2025, 03:08 PM UTC
We’re continuing to see improvements and are getting back to normal operations. We're now monitoring the fix closely.
- resolved Apr 30, 2025, 03:41 PM UTC
This incident has been resolved.
- postmortem May 13, 2025, 10:04 AM UTC
### **Summary** On April 30th 2025, between 15:49 and 16:38 CEST, Voyado Engage experienced an issue that caused delays and failures in API calls. The issue was traced to a problem during a system deployment. Our engineering team identified and resolved the issue quickly, with full service restored by 17:03. ### **Customer Impact** During the incident, a significant number of API calls were delayed or failed. Even successful calls had noticeably slower response times. ### **Root Cause and Mitigation** The issue was caused by a misconfiguration in a system update. This affected how the platform processed internal requests and resulted in service degradation. When the deployment was paused to limit impact, some servers were left running the old version of Engage and could not handle the full traffic load. Our team halted the rollout, developed a fix, and redeployed the updated version. Recovery began at 16:38, with full functionality confirmed by 17:03. ### **Next Steps** To prevent similar issues in the future, we are taking actions to implement improvements to our deployment and monitoring processes and to ensure faster recovery time. We appreciate your patience and understanding, and apologize for any inconvenience. We remain committed to providing a stable and reliable platform experience.