Voyado Outage History

Started Jan 23, 2026, 02:22 PM UTC· resolved Jan 23, 2026, 03:28 PM UTC

1. investigating Jan 23, 2026, 02:22 PM UTC

   the FTP server is currently unreachable. We are investigating.

2. monitoring Jan 23, 2026, 02:45 PM UTC

   The FTP is back up and we are monitoring/investigating root cause.

3. resolved Jan 23, 2026, 03:28 PM UTC

   The FTP is stable and no indication of it returning so resolving the incident for now.

4. postmortem Feb 09, 2026, 05:12 AM UTC

   ## Summary On January 23, approximately between 15:20–16:25 CET, our FTP service experienced an outage after the underlying server became unresponsive following an unexpected spike in resource usage. The service was restored by restarting the affected server. After recovery, we verified stability and completed follow-up maintenance to reduce operational risk. ## Customer impact During the timeframe of the incident, FTP connections and transfers were unavailable for customers using the FTP service. Once recovery actions were completed, the service resumed normal operation. ## Root Cause The incident was triggered by an unexpected CPU spike that caused the server to stall at the system level. As a result, the FTP service stopped responding and the server could not be reached for troubleshooting until it was restarted. ## Mitigation * Restarted the affected server to restore responsiveness and FTP availability. * Confirmed the FTP service was running normally and serving traffic after recovery. * Performed follow-up maintenance to reduce risk. ## Next steps * Improve resilience for the FTP service to reduce the risk of full outages \(e.g., moving toward a more redundant setup\). ‌ We apologize for the disruption of service this caused, and we appreciate your patience while we worked to restore the service.

Read the full incident report →

Started Jan 20, 2026, 10:20 AM UTC· resolved Jan 20, 2026, 11:35 AM UTC

1. investigating Jan 20, 2026, 10:20 AM UTC

   We are currently investigating this issue.

2. identified Jan 20, 2026, 10:30 AM UTC

   We have identified the issue and are preparing a rollback.

3. monitoring Jan 20, 2026, 11:01 AM UTC

   The latencies are improving and we continue to monitor the situation. Not all customers were affected.

4. resolved Jan 20, 2026, 11:35 AM UTC

   This incident has been resolved.

5. postmortem Jan 23, 2026, 02:27 PM UTC

   A release of Elevate introduced a bug in the calculation of promotion statistics, which caused CPU exhaustion leading to performance degradation of some Storefront endpoints. When noticed, the release was aborted and rollback was made to the previous version. The endpoints primarily affected were the landing-page and the product-page. Not all customers were affected.

Read the full incident report →

Started Jan 13, 2026, 08:34 AM UTC· resolved Jan 13, 2026, 11:17 AM UTC

1. identified Jan 13, 2026, 08:34 AM UTC

   We are currently experiencing issues affecting a subset of customers when using Design Studio and emails with Data sources. Emails with an existing Data source are sent as usual, but it is currently not possible to add a Data source to new emails, and Starter modules are not loading. We are actively working to resolve the issue.

2. identified Jan 13, 2026, 09:24 AM UTC

   We have identified the issue and prepared a fix, which is currently being tested and prepared for deployment.

3. resolved Jan 13, 2026, 11:17 AM UTC

   This incident has been resolved.

4. postmortem Jan 14, 2026, 03:28 PM UTC

   ## **Summary** On January 13th, we experienced an issue that temporarily prevented access to the Design Studio for some customers using the Orders module. ## **Customer Impact** Customers with the Orders module enabled and using a specific storage configuration were unable to access data sources and email modules in the Design Studio. Email sending was not affected. ## **Root Cause** The issue was caused by a configuration error that prevented necessary data from loading correctly. ## **Mitigation** We identified the root cause and applied a fix by adjusting the storage setup. Full functionality was restored by 12:14 CET on the same day. ## **Next Steps** To prevent this from happening again, we are improving our internal testing environments and reviewing the need for this specific configuration going forward. We sincerely apologize for the inconvenience this may have caused and appreciate your patience while we worked to resolve the issue.

Read the full incident report →

Started Dec 30, 2025, 11:04 AM UTC· resolved Dec 30, 2025, 11:59 AM UTC

1. investigating Dec 30, 2025, 11:04 AM UTC

   We have noted that one of our servers is having issues which is affecting a subset of our customers with long response time. Investigation ongoing.

2. monitoring Dec 30, 2025, 11:28 AM UTC

   We believe the source of the issue has been found and stopped. Monitoring performance.

3. resolved Dec 30, 2025, 11:59 AM UTC

   The incident is resolved.

4. postmortem Jan 16, 2026, 11:33 AM UTC

   ## Summary On December 30th, a performance issue on one of our servers led to delayed API response times for a subset of customers. The issue was quickly identified and mitigated within approximately one hour. ## Customer Impact The delays affected API responses for customers hosted on one specific server. All other parts of the platform remained fully functional. ## Root Cause The issue was triggered by a large import job that attempted to process a high volume of transactions in a single run. This created a significant number of internal locks in our database, leading to degraded performance. ## Remediation & Prevention Once the problematic job was identified, it was stopped, and a work-around was made to prevent further issues. ## Next Steps We are reviewing the job and do necessary changes to safe-guard how the platform handles high-volume imports, reducing the risk of similar incidents in the future. We sincerely apologize for the inconvenience this caused and appreciate your patience while we worked to resolve the issue. Preventing disruptions like this remains a top priority, and we are committed to learning from this incident to improve platform resilience.

Read the full incident report →

Started Nov 28, 2025, 09:49 AM UTC· resolved Nov 28, 2025, 04:55 PM UTC

1. identified Nov 28, 2025, 09:49 AM UTC

   We are currently investigating an issue affecting the ingestion of statistics, which may result in delays in the display of sent, delivered, open, and click data.

2. identified Nov 28, 2025, 10:41 AM UTC

   We have identified the issue causing delays in the display of statistics in Engage and are currently working to remediate it. At this time, the delay in the ingestion of statistics is approximately one hour.

3. monitoring Nov 28, 2025, 12:58 PM UTC

   We have identified the cause of the issue and the statistics are now returning to normal processing speed. In Engage, two types of statistics are shown: Preliminary statistics, which provide early indicators on a message level. Full statistics, which are generated once statistics for an individual contact has been fully processed. Normally, the preliminary statistics appear much faster, but the disturbance did only affect this quick-to-update data. The full statistics was being processed and ingested within normal parameters. In Engage, preliminary statistics are indicated by a “~” symbol, a “–” placeholder, or a warning on the details page indicating that the statistics are preliminary. When these indications disappear, customers can use email statics for segmentation, even though some data points might still remain as "unknown status".

4. resolved Nov 28, 2025, 04:55 PM UTC

   This incident has been resolved.

5. postmortem Dec 19, 2025, 12:33 PM UTC

   ### Summary On the morning of November 28, we experienced a delay in the systems responsible for processing message statistics in Voyado Engage. While message delivery worked as expected, statistics—such as opens, clicks, and bounces—were shown with a delay of up to 60 minutes in the user interface. ### Customer Impact * Message delivery was not affected. * Users saw delays of up to 60 minutes when reviewing delivery statistics for email send-outs. * Segmentation based on message statistics continued to work, though with a delay. * This affected all customers using Engage to send messages during the morning and early afternoon, the impact was isolated to a delay in viewing the statistics for the messages sent. ### Root Cause The issue was caused by limitations in how our internal message handler for message statistics in particular is scaled during high load. Specifically, the service handling statistics did not automatically increase its capacity as expected due to how scaling logic was configured. ### Mitigation * Manual scaling was applied to increase processing capacity. * Services were restarted to resolve blocks in the processing. * We adjusted the internal message handling strategy to improve throughput. * By mid-afternoon, statistics processing had caught up and delays were resolved. ### Next Steps * Review and update scaling configuration to ensure automatic scaling behaves as expected. We sincerely apologize for the inconvenience this may have caused and appreciate your patience while we worked to resolve the issue.

Read the full incident report →

Started Nov 12, 2025, 09:49 AM UTC· resolved Nov 12, 2025, 12:28 PM UTC

1. investigating Nov 12, 2025, 09:49 AM UTC

   We have reports on issues submitting registration forms - this is being investigated.

2. identified Nov 12, 2025, 10:08 AM UTC

   The issue has been identified an a hotfix is being prepared.

3. identified Nov 12, 2025, 11:29 AM UTC

   Fix is being deployed and issue should be resolved in about 45 min.

4. resolved Nov 12, 2025, 12:28 PM UTC

   Issue has been resolved.

5. postmortem Nov 13, 2025, 02:54 PM UTC

   ## **Summary** On 2025-11-12, an issue was introduced in a morning update that caused registration forms to fail silently on submission for multiple customers. The forms appeared to work but did not process user input. ## **Customer Impact** End-users were unable to submit registration forms or update communication preferences. No error messages were shown, which impacted onboarding and preference management flows. ## **Root Cause** A bug in the released code affected how registration forms handled certain preference fields, causing submissions to fail without visible errors. ## **Mitigation** A fix was identified and deployed to resolve the issue. ## **Next Steps** We are implementing additional testing safeguards and improving our internal processes to detect similar issues earlier. We apologize for the disruption and appreciate your understanding.

Read the full incident report →

Started Nov 07, 2025, 11:42 AM UTC· resolved Nov 07, 2025, 03:54 PM UTC

1. identified Nov 07, 2025, 11:42 AM UTC

   We are currently experiencing a delay in processing events, which impacts features such as Interaction and Target Audience triggers in Automations. Our team is preparing a hotfix to address the issue. We will continue to provide updates as the situation progresses.

2. monitoring Nov 07, 2025, 03:00 PM UTC

   A fix has been implemented, and we are now seeing improvements in event processing times. Features such as Interaction and Target Audience triggers in Automations are gradually returning to normal performance. We continue to monitor the system closely. The underlying root cause is still under investigation, and we’ll provide more information as it becomes available.

3. resolved Nov 07, 2025, 03:54 PM UTC

   The issue causing delays in event processing has now been resolved. All systems are operating as expected. Root cause analysis will be included in Post-Mortem.

4. postmortem Nov 13, 2025, 08:18 AM UTC

   ## Summary On November 7th, an unusually high number of internal events were generated due to a planned feature rollout. This caused delays in processing certain automations. ## Customer Impact Automations triggered by customer interactions or audience targeting were delayed by up to 12 hours. All affected automations were eventually executed. Other parts of the platform operated as normal. ## Root Cause and Mitigation When the new Marketing Groups feature was enabled for all customers, a large volume of events \(hundreds of millions\) was generated in a short time. The system processing these events could not keep up, which resulted in delays for other events in the same internal queue. To restore normal flow, we temporarily cleared out the backlog of non-critical events and performed database synchronizations to ensure data accuracy. Normal operation resumed later the same day. ## Next Steps We are improving our monitoring and alerting to detect similar issues earlier, and we are reviewing our event handling capacity and release procedures to prevent future impact from large-scale rollouts. We apologize for any inconvenience caused and appreciate your patience while we resolved the issue.

Read the full incident report →

Started Oct 14, 2025, 09:08 AM UTC· resolved Oct 14, 2025, 09:45 AM UTC

1. investigating Oct 14, 2025, 09:08 AM UTC

   We are currently investigating this issue.

2. identified Oct 14, 2025, 09:19 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Oct 14, 2025, 09:45 AM UTC

   Incident is resolved. No more errors should be seen.

4. postmortem Oct 14, 2025, 02:32 PM UTC

   **Description** During an A/B test of personalization functionality, an update introduced data that was not compatible with older system versions. This caused search-page errors for some shoppers when their requests were routed between nodes running different versions. The A/B test was disabled promptly, preventing new incompatible data from being written. This action significantly reduced the error rate. Errors continued to occur for a limited number of shoppers who had already been affected, until all older nodes were fully upgraded. * The majority of customers running the test experienced no impact. * Most customers faced none or a few errors. All systems were upgraded to the new version, eliminating the incompatibility and restoring full stability. **Business impact** * Some shoppers attempting searches received error messages, which could disrupt their browsing and purchase flow. * Only customers actively running the personalization A/B test were affected. For most, the number of shoppers impacted was very limited. **Review** * Review of rollout and A/B testing procedures to reduce cross-version risk. * Improve error handling in personalization so that issues cannot cause shopper-facing failures. * Add faster “kill switch” capabilities to disable personalization in case of future incidents.

Read the full incident report →

Started Oct 09, 2025, 09:35 AM UTC· resolved Oct 09, 2025, 10:10 AM UTC

1. investigating Oct 09, 2025, 09:35 AM UTC

   We are currently experiencing a data feed disturbance affecting the Revenue Overview Dashboard. Displayed data may be inaccurate.

2. identified Oct 09, 2025, 09:39 AM UTC

   We believe we have identified the cause of the issue affecting the Revenue Overview Dashboard and are now working on a solution.

3. identified Oct 09, 2025, 09:51 AM UTC

   We are continuing to work on a fix for this issue.

4. resolved Oct 09, 2025, 10:10 AM UTC

   This incident has been resolved.

5. postmortem Oct 13, 2025, 09:37 AM UTC

   ## Summary A background system component failed to start due to outdated software. This caused some data to stop updating for a short period. ## Customer Impact For a limited time, some customers saw outdated or missing data in the Revenue Overview Dashboard. ## Root Cause and Mitigation The issue was caused by an outdated software dependency. The system was updated to the latest version, and we’ve added improved monitoring to detect similar issues earlier in the future.

Read the full incident report →

Started Oct 08, 2025, 10:02 AM UTC· resolved Oct 08, 2025, 01:24 PM UTC

1. identified Oct 08, 2025, 10:02 AM UTC

   We are currently experiencing a delay in how sendout statuses are updated in Engage. Some sendouts may appear to be stuck in Scheduled, even though they have actually been sent as expected. This issue does not affect message delivery, only how the status is shown in the interface. We are working to resolve the issue and will post an update here once it has been fully resolved.

2. resolved Oct 08, 2025, 01:24 PM UTC

   This incident has been resolved.

3. postmortem Nov 06, 2025, 09:57 PM UTC

   ### Summary Between October 7th and October 8th, users could experience delays in the display of message send-out status updates in the Engage interface. While the messages were sent as expected, the visual feedback in the interface did not reflect their actual status. ### Customer Impact During the incident, message send-out status and related statistics were delayed in Engage. No messages were lost or failed to send—this was a display issue only. ### Root Cause The delay was caused by an unusually large batch of events from our data processing pipeline that overwhelmed an internal message handler. The large batch occurred due to a change in configuration during refactoring that led to unintentionally re-sending of all historic events. While the initial ingestion layer could handle the load, the downstream system could not process events fast enough, resulting in a backlog. A key process responsible for moving events downstream stopped functioning correctly, and the issue went unnoticed for several hours adding to the backlog and ultimate delay. ### Mitigation * Redundant subscriptions were removed and recreated to clear the backlog. * Processing power was increased for the affected message handler. * The rate of downstream event processing was reduced to alleviate pressure. * Event processing was monitored until the backlog was fully cleared. By the afternoon of October 8th, the system had fully caught up and normal status updates were visible again in Engage. ### Next Steps To reduce the risk of similar issues occurring in the future, we are implementing the following improvements: * Improving autoscaling for the internal message handler system * Improving safeguards in our data platform to avoid overwhelming downstream systems * Adding delivery time monitoring to our internal dashboards to detect delays earlier We apologize for the inconvenience this may have caused and appreciate your patience while we worked to resolve the issue.

Read the full incident report →

Started Oct 01, 2025, 10:27 AM UTC· resolved Oct 01, 2025, 01:02 PM UTC

1. identified Oct 01, 2025, 10:27 AM UTC

   We are currently experiencing a disruption affecting customers using the Orders v3 endpoint. A fix is in progress and will be deployed shortly.

2. identified Oct 01, 2025, 10:45 AM UTC

   A hotfix is currently being deployed. The estimated time until the Orders v3 endpoint is available again is approximately 60 minutes.

3. identified Oct 01, 2025, 12:06 PM UTC

   The hotfix deployment is still ongoing.

4. monitoring Oct 01, 2025, 12:23 PM UTC

   The hotfix deployment has been successfully released, and services are back to normal operation.

5. resolved Oct 01, 2025, 01:02 PM UTC

   This incident has been resolved.

6. postmortem Oct 15, 2025, 09:35 AM UTC

   # **Summary** On October 1st, 2025, the Order V3 API was unavailable between 12:00 and 14:30 CEST. The issue was caused by a faulty update intended to handle previously failed orders and led to serialization errors that made the API unresponsive. The problem was identified and a fix was deployed to restore normal operations. # **Customer Impact** A few customers were affected during the incident. Customers using the Order V3 API were unable to place or manage orders through this version of the API during the downtime. Other versions and platform features remained unaffected. # **Root Cause** The issue derived from a serialization error introduced during a fix deployment aimed at resending failed order messages. This error caused all incoming commands to fail, rendering the Order V3 API temporarily unusable. # **Mitigation** The issue was identified and a fix was prepared and deployed. Full functionality was restored by 14:30 CEST, the system was closely monitored to ensure stability. # **Next Steps** We are taking several actions to prevent similar incidents: * Improving test coverage and automated validation for critical API updates. * Enhancing our developer tools to better simulate and detect serialization issues. We apologize for the inconvenience caused and appreciate your patience as we continue to improve the reliability of our platform.

Read the full incident report →

Started Sep 30, 2025, 07:21 AM UTC· resolved Sep 30, 2025, 11:40 AM UTC

1. investigating Sep 30, 2025, 07:21 AM UTC

   We have identified an issue impacting email sendouts. We are actively investigating and will provide updates on this page as soon as more information is available.

2. monitoring Sep 30, 2025, 08:02 AM UTC

   A hotfix has been deployed and shows promising results, but unfortunately many messages are still delayed and currently being processed.

3. monitoring Sep 30, 2025, 08:39 AM UTC

   The incident has been resolved and the deployment was successful. Email sendouts are functioning normally again, but a number of delayed messages are still being processed and delivered.

4. monitoring Sep 30, 2025, 09:41 AM UTC

   We are continuing to process and deliver the emails that were delayed due to the earlier incident.

5. resolved Sep 30, 2025, 11:40 AM UTC

   This incident has been resolved. We are continuing to process and deliver the final messages that were delayed due to the incident.

6. postmortem Oct 13, 2025, 12:59 PM UTC

   ## Summary On September 30th, Engage experienced a temporary disruption in email send-outs due to an error in newly deployed code. The issue was quickly identified, and a hotfix was deployed. While no messages were lost, some email messages were delayed as the system worked through a backlog. ## Customer Impact Majority of customers were potentially affected. Most experienced delays in automation emails. In a few cases, delays were up to 8 hours, but the majority of impacted messages were delivered within 3–4 hours. No SMS or priority messages were affected. ## Root Cause The disruption was caused by an internal misconfiguration in how the platform determined where to store and retrieve data for email messages. When a new code deployment corrected this configuration, it unintentionally disrupted messages that had already been processed using the old setup. This led to messages being queued and not sent. ## Mitigation After identifying the issue, a hotfix was deployed to prevent further impact. Additional work was done throughout the day to manually resend affected messages and ensure all queued messages were successfully delivered. ## Next Steps To reduce the risk of similar issues in the future, we are: * Implementing automated code checks that will detect and block misconfigurations before deployment. * Improving internal processes for incident handling and maintenance tooling, including procedures to quickly resending impacted messages. We apologize for any inconvenience caused and appreciate your patience while we resolved the issue.

Read the full incident report →

Started Sep 28, 2025, 08:34 AM UTC· resolved Sep 28, 2025, 10:14 AM UTC

1. investigating Sep 28, 2025, 08:34 AM UTC

   We are currently experiencing delays with sendouts affecting a subset of customers. We are investigating and will provide updates as soon as we have more information.

2. monitoring Sep 28, 2025, 09:13 AM UTC

   We identified the issue that was causing message delays for a subset of customers. The affected messages are now being processed, and we will continue to monitor closely until all messages have been sent.

3. resolved Sep 28, 2025, 10:14 AM UTC

   The delayed messages have been processed and we are back to normal operations.

4. postmortem Oct 13, 2025, 12:56 PM UTC

   ## Summary On September 28th, Engage experienced delays in sending bulk email messages. The issue was caused by part of our internal email processing setup not operating as expected. The delay was resolved the same day, and no messages were lost. ## Customer Impact Bulk emails for a minor sub-set of customers were delayed by up to 3 hours. Priority messages, including automation emails and SMS, were not affected and were delivered as normal. ## Root Cause Three of the internal services responsible for processing emails were not active due to a system update on September 27th. This caused them to restart without automatically reactivating the needed components. Combined with a high traffic in the system, this led to a backlog in the email messaging pipeline. ## Mitigation The inactive services were redeployed, restoring full processing capacity. Once this was done, the message queue was quickly processed, and operations returned to normal. ## Next Steps To prevent similar issues in the future, we are: * Updating our deployment pipeline to ensure the required components are automatically restarted after updates. * Introducing additional monitoring and alerts to detect inactive or underperforming internal services sooner. We appreciate your understanding and patience while we worked through this issue and apologize for any inconvenience.

Read the full incident report →

Started Sep 26, 2025, 08:48 AM UTC· resolved Sep 26, 2025, 11:20 AM UTC

1. investigating Sep 26, 2025, 08:48 AM UTC

   Outages with an upstream service provider have caused more delays with the analytics data.

2. identified Sep 26, 2025, 08:57 AM UTC

   Our investigation is ongoing. So far, we have not seen evidence of customer impact.

3. resolved Sep 26, 2025, 11:20 AM UTC

   The issue was automatically resolved earlier this morning. Any potential customer impact was already cleared before we began investigating.

Read the full incident report →

Started Sep 23, 2025, 12:39 PM UTC· resolved Sep 25, 2025, 01:56 PM UTC

1. identified Sep 23, 2025, 12:39 PM UTC

   We are currently experiencing issues with some of our analytics data. As a result, statistics shown in reports may be incorrect or incomplete starting from yesterday (Monday). No data has been lost, and the underlying data will be reflected accurately once the issue is resolved. We are working to fix the problem and will provide updates at a later point.

2. identified Sep 23, 2025, 01:28 PM UTC

   Analytic data creation for the affected entities should now be working as expected again. However, there are still gaps in analytics data from yesterday. Our team is actively backfilling this missing data to ensure reports are accurate. We expect this process to be completed during the day tomorrow. No data has been lost, and once backfilling is done, all statistics will be fully restored.

3. identified Sep 24, 2025, 02:41 PM UTC

   The process of restoring the missing analytics data is still ongoing and taking longer than expected. Once the data has been fully restored, we will trigger a re-aggregation to ensure the affected reports display the correct statistics. We expect this process to be completed tomorrow and will continue to share updates as progress is made

4. identified Sep 25, 2025, 09:50 AM UTC

   The restoration of the missing analytics data has now been completed. Reports will show the correct statistics once data has been fully re-aggregated, which is expected to be reflected from this afternoon (approximately 13:00–16:00 CEST). Thank you for your patience.

5. resolved Sep 25, 2025, 01:56 PM UTC

   The incident has now been resolved, and analytics data has been fully updated. Reports are displaying correct and complete statistics again. Thank you for your patience throughout this incident.

Read the full incident report →

Started Sep 23, 2025, 10:28 AM UTC· resolved Sep 23, 2025, 12:01 PM UTC

1. investigating Sep 23, 2025, 10:28 AM UTC

   We are currently investigating reports of general slowness and degraded performance across the platform. Users may experience unusually long loading times in the web application and APIs, and this may also impact processing times for automations and sendouts. Our team is working to identify the root cause and will provide an update as soon as more information is available. Thank you for your patience.

2. monitoring Sep 23, 2025, 10:49 AM UTC

   We have taken actions that appear to have improved overall platform performance. Loading times in the web application and processing times for automations and sendouts are returning to normal levels. We are continuing to monitor the situation closely to ensure stability and will provide further updates as needed.

3. resolved Sep 23, 2025, 12:01 PM UTC

   We have monitored the platform closely following the earlier performance issues, and all systems have remained stable. Loading times and processing are back to normal levels. While we are still investigating the underlying root cause to prevent recurrence, the immediate issue has been resolved. Thank you for your patience during this incident.

Read the full incident report →

Started Sep 17, 2025, 07:41 AM UTC· resolved Sep 17, 2025, 09:21 AM UTC

1. investigating Sep 17, 2025, 07:41 AM UTC

   We are currently experiencing delays with sendouts. A subset of customers may also be experiencing degraded API performance. We are investigating and will provide updates as soon as we have more information.

2. investigating Sep 17, 2025, 08:08 AM UTC

   Our teams are fully engaged and continue working with high priority to resolve the situation. However, the issue is still ongoing and some users may continue to experience problems. We’ll share further updates as soon as we have more information.

3. identified Sep 17, 2025, 08:22 AM UTC

   We have identified the issue and taken remedial action. Initial improvements have been observed, with API performance and message delays showing signs of recovery. We're currently working on an additional fix to ensure long-term stability and are closely monitoring the situation before confirming full return to normal operations.

4. identified Sep 17, 2025, 08:59 AM UTC

   A hotfix has been prepared and is currently being deployed. We will share an update as soon as we know the result and the effect it has.

5. resolved Sep 17, 2025, 09:21 AM UTC

   We have returned to normal API operations, and new messages are being sent as expected. The messages that were queued and delayed due to the incident are now being processed. A follow-up post-mortem incident report will be shared.

6. postmortem Sep 25, 2025, 09:44 AM UTC

   ### Summary On the morning of September 17th, the Engage platform experienced degraded performance primarily affecting messaging and APIs. The root cause was traced to a scenario triggered by the intentional disabling of messaging functionality for a specific tenant. This caused an unusually high load on our infrastructure, causing ripple effects in parts of the system affecting other tenants as well. ### Customer Impact * **Messaging delays** of 10–30 minutes for most customers. One tenant experienced longer delays \(up to 60 minutes\) and required minor manual intervention for a small number of messages. * **APIs affected**: CreateContact, GetContact, and UpdateContact showed poor response times for a sub-set of customers. No messages were lost, and all systems were fully recovered by 11:10 CEST. ### Root Cause The incident stemmed from disabling messaging functionality for a specific tenant in combination with messaging activity for that tenant. This generated a lot of activity in the system among other triggering a poorly optimized database query. The query caused significant load on our database servers, leading to cascading delays across queues and APIs. ### Mitigation * Redeployment of services to free up resources * Actions to unblock queued up events * A hotfix was deployed to optimize an identified complex query and reduce system strain. ### Next Steps To prevent similar issues we are working on improvements for: * Query handling for stopped messages will be further optimized. * Enhanced monitoring and tooling around internal message handling usage and query execution are being introduced. We appreciate your patience and understanding, and apologize for any inconvenience. Please reach out if you have any questions or need further clarification.

Read the full incident report →

Started Sep 14, 2025, 11:30 AM UTC· resolved Sep 14, 2025, 01:45 PM UTC

1. resolved Sep 15, 2025, 02:37 PM UTC

   We experienced an issue affecting a small subset of clusters where product results were not ordered with the expected relevance. The incident occurred between approximately 11:30 UTC and 13:45 UTC. The issue has been resolved, and relevance order is now restored for all affected clusters. No further impact is expected.

Read the full incident report →

Started Aug 20, 2025, 12:29 PM UTC· resolved Aug 20, 2025, 03:17 PM UTC

1. investigating Aug 20, 2025, 12:29 PM UTC

   We are currently investigating a potential disruption. At this stage, information is limited, further updates will be shared as soon as possible.

2. monitoring Aug 20, 2025, 12:57 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Aug 20, 2025, 01:21 PM UTC

   We are continuing to monitor for any further issues. Current performance remains stable and has been looking good since the implemented fix.

4. resolved Aug 20, 2025, 03:17 PM UTC

   This incident has been resolved.

5. postmortem Aug 26, 2025, 07:19 AM UTC

   **Summary** On August 20th, we experienced an issue affecting connectivity and performance in our API Gateway. The incident was identified and resolved within a short time frame. **Customer impact** During the incident, some customers experienced degraded performance and intermittent connectivity issues when accessing the API Gateway. Other platform functionality remained unaffected. **Root cause** The issue was caused by an SSL certificate deployment that introduced unexpected connectivity and performance problems. **Mitigation** Once identified, the deployment was quickly reverted, which immediately stabilized the platform. A permanent fix has since been implemented, and no further performance degradation has been observed. **Next steps** We have updated our deployment routines and processes to ensure similar issues are avoided in the future.

Read the full incident report →

Started Aug 19, 2025, 08:05 AM UTC· resolved Aug 19, 2025, 02:49 PM UTC

1. identified Aug 19, 2025, 08:05 AM UTC

   We have identified increased response times on our Contacts API. Users may experience slow response times and potential timeouts. Our team is actively working on deploying a hotfix as soon as possible to resolve the issue.

2. monitoring Aug 19, 2025, 09:55 AM UTC

   A fix has been implemented and we are monitoring the results.

3. identified Aug 19, 2025, 11:22 AM UTC

   Unfortunately our initial hotfix did not fully resolve our API response times. We are currently deploying a new hotfix to resolve this issue.

4. monitoring Aug 19, 2025, 01:01 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Aug 19, 2025, 02:49 PM UTC

   We have been monitoring the performance closely and can confirm that services are now running as expected. We apologize for any inconvenience caused during this incident, and can confirm that it has been resolved.

6. postmortem Aug 27, 2025, 12:30 PM UTC

   ## Summary On August 19th, Voyado Engage experienced degraded performance on the Contact API due to an issue introduced in a recent release. ## Customer Impact Customers using the Contact API may have experienced slower response times. Other parts of the platform remained fully operational. ## Root Cause A code change led to unintended system behavior that affected backend performance. The issue was related to a specific component used to detect changes in contact data. ## Mitigation The problematic code was disabled through a targeted hotfix, which restored normal performance. ## Next steps We are improving safeguards around this component and enhancing our deployment checks to prevent similar issues in the future. We are sorry for the disruption this caused and appreciate your patience while we worked to resolve the issue.

Read the full incident report →

Started Aug 10, 2025, 12:01 PM UTC· resolved Aug 10, 2025, 04:46 PM UTC

1. identified Aug 10, 2025, 12:01 PM UTC

   We have identified delays in delivery statistics affecting a subset of our customers since 10.19 today. According to our SMS provider, delays in SMS deliveries may also be experienced. The disruption is due to an ongoing incident at one of our SMS providers. You can follow the provider’s status here: https://status.sinch.com/

2. monitoring Aug 10, 2025, 01:09 PM UTC

   Our SMS provider has implemented a fix, and we have since observed that the delay in delivery statistics for sendouts is resolving.

3. resolved Aug 10, 2025, 04:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 01, 2025, 01:26 PM UTC· resolved Aug 01, 2025, 03:37 PM UTC

1. investigating Aug 01, 2025, 01:26 PM UTC

   We are investigating indications of a potential issue affecting the processing of email sendouts.

2. investigating Aug 01, 2025, 02:51 PM UTC

   We have identified that the current delays are only affecting emails sent through automations. Our team is actively working on a solution to resolve this as quickly as possible. We will continue to share updates here as soon as new information becomes available.

3. monitoring Aug 01, 2025, 03:19 PM UTC

   We have implemented a fix that is showing very promising results. The queue of delayed messages is rapidly decreasing. We will continue to monitor the progress until the processing times are back to normal.

4. resolved Aug 01, 2025, 03:37 PM UTC

   The delay affecting email sendouts in automations has been resolved, and processing speeds have returned to normal. We will continue to monitor the email queues to ensure ongoing stability.

5. postmortem Aug 26, 2025, 06:22 AM UTC

   ## Summary On August 1st, approximately between 14:25–17:30 CEST, Voyado Engage experienced issues where email send-outs triggered by Automation workflows were delayed. The issue was caused by a combination of several factors leading to failure in processing for the internal messages handler, halting workflow execution leading to a delay in email send-outs. ## Customer Impact Tenants with Automation workflows triggering email send-outs during the incident window were affected. While no messages were lost, all send-outs were delayed until the issue was resolved and normal operations resumed. ## Root Cause Our investigations leads us to the conclusion that the issue was caused by a combination of several factors where services in charge of processing internal messaging becomes overloaded and unable to process their commands. This caused a full stop in the execution of Automations. A record-high number of messages were in queue, putting an immediate stress to the system and exhausting system capacity. These services had not been fully restarted for a period longer than usual, which may have contributed to degraded performance. ## Mitigation The issue was resolved by incrementally restarting the affected services. As they were brought back online, message processing resumed and queues cleared automatically. No manual reprocessing was required and all delayed send-outs were successfully delivered. ## Next Steps We have reviewed and improved monitoring of queues for internal messaging, added monitoring to understand more on infrastructure health during similar events in the future and are continuously working on improving performance to ensure stability and quality. We apologize for the inconvenience this may have caused and appreciate your patience as we worked to restore normal operations.

Read the full incident report →

Started Jul 22, 2025, 04:35 PM UTC· resolved Jul 22, 2025, 06:47 PM UTC

1. investigating Jul 22, 2025, 06:26 PM UTC

   We are currently investigating this issue.

2. resolved Jul 22, 2025, 06:47 PM UTC

   This incident has been resolved.

3. postmortem Jul 24, 2025, 07:04 AM UTC

   A separate service from a third party/vendor is used to host the Elevate application. Due to an error in their invoicing service the Elevate application was suspended.

Read the full incident report →

Started Jul 02, 2025, 02:31 PM UTC· resolved Jul 02, 2025, 06:51 PM UTC

1. investigating Jul 02, 2025, 02:31 PM UTC

   We are currently experiencing service disruptions due to an issue with an external provider. This may affect the availability or performance of contact lookups for the subset of customers which are using this feature. We are monitoring the situation closely and will provide updates as more information becomes available.

2. investigating Jul 02, 2025, 03:47 PM UTC

   This issue is still being investigated by our provider. Current status and more detailed updates can be followed here: https://status.dnb.com/. We will keep you informed about the development as we learn more.

3. resolved Jul 02, 2025, 06:51 PM UTC

   The affected service is restored back to normal and contact lookups are now fully operational again.

Read the full incident report →

Started Jun 26, 2025, 09:35 AM UTC· resolved Jun 26, 2025, 02:47 PM UTC

1. identified Jun 26, 2025, 09:35 AM UTC

   There's a current decrease in the amount of Abandoned cart automations being triggered. A fix is being deployed.

2. identified Jun 26, 2025, 10:45 AM UTC

   We’re seeing that this issue is affecting the entire tracking pipeline, including abandoned cart and products of interest. A fix is currently being deployed and we are monitoring the progress. We sincerely apologize for any inconvenience this may cause.

3. identified Jun 26, 2025, 11:01 AM UTC

   The fix currently being deployed is expected to be fully implemented in approximately 2 hours. Thank you for your patience.

4. identified Jun 26, 2025, 01:01 PM UTC

   Abandoned cart automations are now working as expected. We are still working on a fix for Products of Interest.

5. resolved Jun 26, 2025, 02:47 PM UTC

   This incident has been resolved.

6. postmortem Jul 04, 2025, 09:41 AM UTC

   ## Summary Between June 23 and June 26, 2025, some customers experienced incorrect triggering of Abandoned Cart \(AC\) emails, where messages were sent despite their carts being empty. This issue was caused by an error in the logic that determines the most recent cart activity within our automation pipeline. While a fix was deployed quickly, the resolution was delayed due to the time required to reprocess historical data through our streaming systems. ## Customer Impact · Some customers received Abandoned Cart emails that did not reflect their actual shopping activity. · The Product of Interest \(POI\) automation experienced delays and was temporarily paused during recovery efforts. · Email reliability and accuracy were temporarily impacted for a subset of users. ## Root Cause and Mitigation The incident was caused by **non-deterministic logic** in the `abandoned_and_poi_pipeline` that selects the latest cart tracking event. This led to outdated or incorrect data being interpreted as current, which in turn triggered abandoned cart emails for empty carts. A hotfix correcting this logic was deployed to production on June 25. However, because the streaming job had to be restarted from a historical point \(June 17\), the large volume of data delayed full recovery. The pipeline did not catch up to the most recent checkpoint within the expected time frame, requiring manual intervention with larger compute clusters. The Abandoned Cart flow was restored by the afternoon of June 26, followed by the POI flow shortly thereafter. ## Next Steps To prevent recurrence and improve incident response, we are taking the following actions: · Refactoring the event-fetching logic to ensure deterministic and reliable behavior. · Decoupling the Abandoned Cart and Product of Interest workflows to enable more targeted incident handling. · Improving on-call documentation and training to ensure faster, more informed decisions during incident resolution. · Enhancing our data pipeline observability and scaling strategies to better handle reprocessing of large datasets.

Read the full incident report →