Simon Data incident
Simon Mail event collection degraded performance
Simon Data experienced a minor incident on February 23, 2024 affecting Simon Mail (message sending), lasting 6h 43m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 23, 2024, 07:44 PM UTC
Due to an outage with our mail transfer agent (MTA) partner, email events like opens and clicks may currently be underreported for any sends after 1PM EST 2/22/24. A fix has been identified and those events will be backfilled once the fix is in place. The fix is expected to go into place in the next few hours, then replay work will begin. Will will update here with more info as it comes in.
- monitoring Feb 23, 2024, 09:18 PM UTC
From our partner: "We have verified that the fix is working as expected and customers are seeing generation of live open and click events. We are actively finalizing testing on the replays and anticipate starting the recovery of historical open and click events within the next few hours. " It is expected that events will be replayed and data might not be up to date as they "plan to replay 1 hour of data at a time, starting from the latest hour and working up to present time, with a pause of 5 minutes between each hour of data." We are continuing to monitor the fix and will update this page with any further information.
- monitoring Feb 23, 2024, 09:22 PM UTC
We are continuing to monitor for any further issues.
- resolved Feb 24, 2024, 02:28 AM UTC
This incident has been resolved. Our MTA partner started replaying historical missed data at 8:30ET. Customers will begin to see missed events delivered to their webhooks and continuing until all data has been replayed. We plan to replay 1 hour of open and click events from 2/22 1800UTC through (and including) 2/23 19:00UTC with a 5 minute pause between each hour of data.