Voyado incident

[Engage] Delays in Processing Message Statistics

Minor Resolved View vendor source →

Voyado experienced a minor incident on November 28, 2025 affecting Messaging, lasting 7h 6m. The incident has been resolved; the full update timeline is below.

Started
Nov 28, 2025, 09:49 AM UTC
Resolved
Nov 28, 2025, 04:55 PM UTC
Duration
7h 6m
Detected by Pingoru
Nov 28, 2025, 09:49 AM UTC

Affected components

Messaging

Update timeline

  1. identified Nov 28, 2025, 09:49 AM UTC

    We are currently investigating an issue affecting the ingestion of statistics, which may result in delays in the display of sent, delivered, open, and click data.

  2. identified Nov 28, 2025, 10:41 AM UTC

    We have identified the issue causing delays in the display of statistics in Engage and are currently working to remediate it. At this time, the delay in the ingestion of statistics is approximately one hour.

  3. monitoring Nov 28, 2025, 12:58 PM UTC

    We have identified the cause of the issue and the statistics are now returning to normal processing speed. In Engage, two types of statistics are shown: Preliminary statistics, which provide early indicators on a message level. Full statistics, which are generated once statistics for an individual contact has been fully processed. Normally, the preliminary statistics appear much faster, but the disturbance did only affect this quick-to-update data. The full statistics was being processed and ingested within normal parameters. In Engage, preliminary statistics are indicated by a “~” symbol, a “–” placeholder, or a warning on the details page indicating that the statistics are preliminary. When these indications disappear, customers can use email statics for segmentation, even though some data points might still remain as "unknown status".

  4. resolved Nov 28, 2025, 04:55 PM UTC

    This incident has been resolved.

  5. postmortem Dec 19, 2025, 12:33 PM UTC

    ### Summary On the morning of November 28, we experienced a delay in the systems responsible for processing message statistics in Voyado Engage. While message delivery worked as expected, statistics—such as opens, clicks, and bounces—were shown with a delay of up to 60 minutes in the user interface. ### Customer Impact * Message delivery was not affected. * Users saw delays of up to 60 minutes when reviewing delivery statistics for email send-outs. * Segmentation based on message statistics continued to work, though with a delay. * This affected all customers using Engage to send messages during the morning and early afternoon, the impact was isolated to a delay in viewing the statistics for the messages sent. ### Root Cause The issue was caused by limitations in how our internal message handler for message statistics in particular is scaled during high load. Specifically, the service handling statistics did not automatically increase its capacity as expected due to how scaling logic was configured. ### Mitigation * Manual scaling was applied to increase processing capacity. * Services were restarted to resolve blocks in the processing. * We adjusted the internal message handling strategy to improve throughput. * By mid-afternoon, statistics processing had caught up and delays were resolved. ### Next Steps * Review and update scaling configuration to ensure automatic scaling behaves as expected. We sincerely apologize for the inconvenience this may have caused and appreciate your patience while we worked to resolve the issue.