Dyopath incident

Delayed Processing of Emails and Workflows

Resolved after 5 hours 22 minutes, affecting Email Engine and Workflows, on January 30, 2025.

resolved Jan 30, 2025, 10:28 PM UTC

Rolling restart has completed and we have confirmed that all Emails and Workflows are processing as expected.

Started
Jan 30, 2025, 05:05 PM UTC
Resolved
Jan 30, 2025, 10:28 PM UTC
Duration
5h 22m
Detected by Pingoru
Jan 30, 2025, 05:05 PM UTC

Affected components

Email EngineWorkflows

Update timeline

  1. monitoring Jan 30, 2025, 05:05 PM UTC

    We were made aware that email processing and some other backend processes in Helix were queueing for an extended period of time this morning. Some were not being processed for up to 40 minutes. We've engaged with BMC support and queued emails and backend processes appear to be processing as expected. We will continue to monitor this issue to confirm resolution.

  2. monitoring Jan 30, 2025, 06:43 PM UTC

    We are continuing to monitor this issue. The processing delays have improved but we are still experiencing longer than usual processing times. We are still engaged with BMC support to fully resolve this issue.

  3. monitoring Jan 30, 2025, 07:41 PM UTC

    BMC will be performing a rolling restart of production servers to address the ongoing issues. No service interruption is expected, however users may receive some session errors if their session is connected to the server being restarted at that time. A simple refresh of the page for the user should resolve the error they may receive.

  4. resolved Jan 30, 2025, 10:28 PM UTC

    Rolling restart has completed and we have confirmed that all Emails and Workflows are processing as expected.

Your product runs on more than Dyopath

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →