Voyado experienced a major incident on October 1, 2025 affecting API, lasting 2h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 01, 2025, 10:27 AM UTC
We are currently experiencing a disruption affecting customers using the Orders v3 endpoint. A fix is in progress and will be deployed shortly.
- identified Oct 01, 2025, 10:45 AM UTC
A hotfix is currently being deployed. The estimated time until the Orders v3 endpoint is available again is approximately 60 minutes.
- identified Oct 01, 2025, 12:06 PM UTC
The hotfix deployment is still ongoing.
- monitoring Oct 01, 2025, 12:23 PM UTC
The hotfix deployment has been successfully released, and services are back to normal operation.
- resolved Oct 01, 2025, 01:02 PM UTC
This incident has been resolved.
- postmortem Oct 15, 2025, 09:35 AM UTC
# **Summary** On October 1st, 2025, the Order V3 API was unavailable between 12:00 and 14:30 CEST. The issue was caused by a faulty update intended to handle previously failed orders and led to serialization errors that made the API unresponsive. The problem was identified and a fix was deployed to restore normal operations. # **Customer Impact** A few customers were affected during the incident. Customers using the Order V3 API were unable to place or manage orders through this version of the API during the downtime. Other versions and platform features remained unaffected. # **Root Cause** The issue derived from a serialization error introduced during a fix deployment aimed at resending failed order messages. This error caused all incoming commands to fail, rendering the Order V3 API temporarily unusable. # **Mitigation** The issue was identified and a fix was prepared and deployed. Full functionality was restored by 14:30 CEST, the system was closely monitored to ensure stability. # **Next Steps** We are taking several actions to prevent similar incidents: * Improving test coverage and automated validation for critical API updates. * Enhancing our developer tools to better simulate and detect serialization issues. We apologize for the inconvenience caused and appreciate your patience as we continue to improve the reliability of our platform.