Voyado incident

Engage - Longer response times in API

Resolved after 21 hours 54 minutes, affecting API, on October 19, 2026.

postmortem Oct 22, 2026, 08:09 AM UTC

### Summary On October 19, between approximately 11:05 and 12:00 CEST, the Voyado Engage APIs experienced degraded performance and some API requests failed, this was due to an infrastructure issue affecting our cloud provider. The issue caused increased response times across our API services, which were further amplified by an internal traffic management mechanism. We identified the contributing factors and took action to restore normal performance. ### Customer impact During the incident, customers experienced slower response times when making requests to the Voyado Engage APIs. Some customers also experienced failed requests, with temporary errors \(503\). Performance was back to normal at about 12:00 CEST. Other platform functionality was confirmed not to be affected. ### Root cause The incident was initially triggered by an infrastructure issue at our cloud provider, which caused reduced performance across multiple services simultaneously. This triggered an internal traffic management mechanism designed to balance load and isolate unusually slow API traffic to protect overall platform performance. The mechanism did not manage the situation as intended which caused delays and some failed API requests for affected customers. ### Mitigation As part of our mitigation efforts, we increased the capacity available to handle the redirected API traffic. Alongside the recovery of the underlying cloud infrastructure, this helped restore normal API performance by approximately 12:00 CEST on October 19. We continued monitoring the platform to ensure stability before formally closing the incident on October 20. ### Next steps To reduce the risk of similar incidents, we are reviewing and taking appopriate actions of the following: * Review and enhance how our traffic management mechanism responds to platform-wide performance degradation. * Expand the capacity available to handle redirected API traffic to improve resilience during unexpected load. * Improve our ability to detect unusual traffic patterns and infrastructure-related performance issues sooner. We remain committed to improving the reliability and resilience of Voyado Engage and appreciate your patience and understanding. We apologize for the disruption this caused.

Started
Oct 19, 2026, 09:49 AM UTC
Resolved
Oct 20, 2026, 07:43 AM UTC
Duration
21h 54m
Detected by Pingoru
Oct 19, 2026, 09:49 AM UTC

Affected components

API

Update timeline

  1. investigating Oct 19, 2026, 09:49 AM UTC

    We are currently seeing longer response times in our APIs - investigation ongoing.

  2. monitoring Oct 19, 2026, 10:22 AM UTC

    Changes was made to remediate the slowness, and things are looking better since 11:59 CEST.

  3. resolved Oct 20, 2026, 07:43 AM UTC

    Monitoring has shown stabilized operational status of the response time. We are conducing a post mortem to understand what was the root cause and will update with an incident report.

  4. postmortem Oct 22, 2026, 08:09 AM UTC

    ### Summary On October 19, between approximately 11:05 and 12:00 CEST, the Voyado Engage APIs experienced degraded performance and some API requests failed, this was due to an infrastructure issue affecting our cloud provider. The issue caused increased response times across our API services, which were further amplified by an internal traffic management mechanism. We identified the contributing factors and took action to restore normal performance. ### Customer impact During the incident, customers experienced slower response times when making requests to the Voyado Engage APIs. Some customers also experienced failed requests, with temporary errors \(503\). Performance was back to normal at about 12:00 CEST. Other platform functionality was confirmed not to be affected. ### Root cause The incident was initially triggered by an infrastructure issue at our cloud provider, which caused reduced performance across multiple services simultaneously. This triggered an internal traffic management mechanism designed to balance load and isolate unusually slow API traffic to protect overall platform performance. The mechanism did not manage the situation as intended which caused delays and some failed API requests for affected customers. ### Mitigation As part of our mitigation efforts, we increased the capacity available to handle the redirected API traffic. Alongside the recovery of the underlying cloud infrastructure, this helped restore normal API performance by approximately 12:00 CEST on October 19. We continued monitoring the platform to ensure stability before formally closing the incident on October 20. ### Next steps To reduce the risk of similar incidents, we are reviewing and taking appopriate actions of the following: * Review and enhance how our traffic management mechanism responds to platform-wide performance degradation. * Expand the capacity available to handle redirected API traffic to improve resilience during unexpected load. * Improve our ability to detect unusual traffic patterns and infrastructure-related performance issues sooner. We remain committed to improving the reliability and resilience of Voyado Engage and appreciate your patience and understanding. We apologize for the disruption this caused.

Your product runs on more than Voyado

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →