Simon Data incident

Degraded Campaign Execution

Minor Resolved View vendor source →

Simon Data experienced a minor incident on December 19, 2022 affecting Simon Event Trigger Flows and Simon Data Syncs, lasting 1h 45m. The incident has been resolved; the full update timeline is below.

Started
Dec 19, 2022, 10:03 PM UTC
Resolved
Dec 19, 2022, 11:48 PM UTC
Duration
1h 45m
Detected by Pingoru
Dec 19, 2022, 10:03 PM UTC

Affected components

Simon Event Trigger FlowsSimon Data Syncs

Update timeline

  1. identified Dec 19, 2022, 10:03 PM UTC

    At 3:13PM EST the infrastructure that executes campaigns began experiencing degradation impacting flows, data feed exports, and our legacy journeys product. Our team identified the symptom and addressed it, but there was a subsequent recurrence which is now being addressed via a deeper infrastructural remediation. We expect delays of messaging to be not longer than 3 hours in total before all campaigns are back on track.

  2. monitoring Dec 19, 2022, 10:59 PM UTC

    At 5:31PM EST a fix for the infrastructure that executes campaigns was implemented. The new infrastructure is now live and processing jobs in a normal, stable state. We expect all campaign sends that were delayed to be recovered in say an hour. Newly sent flows are processing normally as expected

  3. resolved Dec 19, 2022, 11:48 PM UTC

    This incident has been resolved. All flows have been successfully recovered.