FatTail incident

Users may see non-time-based workflow items failing to fire where Field Value Filter(s) are used

Notice Resolved View vendor source →

FatTail experienced a notice incident on April 24, 2023 affecting AdBook+ Order Management, lasting 6h 53m. The incident has been resolved; the full update timeline is below.

Started
Apr 24, 2023, 07:46 PM UTC
Resolved
Apr 25, 2023, 02:40 AM UTC
Duration
6h 53m
Detected by Pingoru
Apr 24, 2023, 07:46 PM UTC

Affected components

AdBook+ Order Management

Update timeline

  1. identified Apr 24, 2023, 07:46 PM UTC

    We have identified an issue where non-time-based workflow items may fail to fire if Field Value Filter(s) are enabled. The incident was caused by a fix for time-based workflow items included in the release of v3.71.01 deployed to AdBook+ production sites on Friday, April 21st. We are currently assessing a fix and will update this incident with timing as soon as it is available. Please contact us at support.fattail.com for questions or concerns. FatTail Support

  2. identified Apr 24, 2023, 09:59 PM UTC

    A fix has been identified and is being tested. Our goal is to deploy an update at 9pm ET today, April 24th. Please reach out via support.fattail.com with any questions or concerns. Thank you for your patience while we work to resolve the issue. FatTail Support

  3. resolved Apr 25, 2023, 02:40 AM UTC

    This issue was resolved with the release of v3.71.02. Workflow tasks may need to be re-triggered according to your unique workflow settings. Please contact us for any follow up questions or support with your workflow. FatTail Support

  4. postmortem May 10, 2023, 07:37 PM UTC

    ### **Summary of Impact** A defect was introduced to AdBookOMS production sites which prevented non-time-based workflow items configured to use Field Value Filter\(s\) from executing. The defect was introduced on Friday, April 21st, 2023 at 9pm ET. FatTail was made aware of the issue through client reports to our Support Center at approximately 9am ET on Monday, April 24th. The engineering team was immediately engaged and worked to take corrective action by developing, testing, and deploying a fix to affected production sites at 9pm ET on Monday, April 24th. During the time that the defect was present, non-time-based workflow tasks which are filtered on field values did not execute. FatTail provided reporting to identify campaigns which may have been affected so that clients could make the necessary adjustments to campaign statuses and ‘catch up’ on tasks which may have been missed in their workflow execution. ### **Root Cause** The incident was caused by a fix for time-based workflow items included in the release of v3.71.01 deployed to AdBook\+ production sites on Friday, April 21st, 2023. ### **Mitigation** FatTail QA team have taken steps to audit and remediate both manual and automated test cases to ensure that all workflow trigger behavior is verified and working as expected ahead of any version upgrades to production sites.