Iterable incident

Event-Triggered Journeys Delays Ingesting New Users

Major Resolved View vendor source →

Iterable experienced a major incident on December 31, 2024 affecting Global Web Application, lasting 3h 55m. The incident has been resolved; the full update timeline is below.

Started
Dec 31, 2024, 02:29 PM UTC
Resolved
Dec 31, 2024, 06:25 PM UTC
Duration
3h 55m
Detected by Pingoru
Dec 31, 2024, 02:29 PM UTC

Affected components

Global Web Application

Update timeline

  1. identified Dec 31, 2024, 02:29 PM UTC

    Summary: Event-Triggered Journeys Delays Ingesting New Users. The issue originated from errors in the workflow-entrance-trigger pods, causing a significant backlog in processing. There is no impact to Scheduled Journeys and API Triggered Journeys . Actions Taken The workflow-entrance-trigger service was updated to the latest version, and additional pods were scaled up to process the backlog faster. The deployment resolved the issue, and error rates dropped significantly. Current Status The errors we were experiencing have been fixed since 5AM PST, now we're just monitoring the backlog as it drains. For 99% of clients, the backlog has drained completely, there are a few stragglers with small backlogs Next Steps Engineers will continue monitoring error rates and ensure the backlog clears entirely. Follow-up tasks include setting up error rate monitoring and addressing journey-specific issues to prevent recurrence.

  2. identified Dec 31, 2024, 04:11 PM UTC

    We are continuing to work on a fix for this issue.

  3. resolved Dec 31, 2024, 06:25 PM UTC

    This issue is now resolved and the backlog is completely drained. Event trigger journey's as of 7:25 AM PST are back to normal and processing as expected. If you still have any further questions please reach out to [email protected].