sendwithus experienced a major incident on February 15, 2019 affecting API and Email Delivery, lasting 1h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 15, 2019, 11:15 PM UTC
Some customers are experiencing issues with activating drip campaigns and running legacy segmentation. The Sendwithus engineering team is currently investigating. API triggered emails are still sending normally.
- monitoring Feb 15, 2019, 11:54 PM UTC
We've identified the issue, deployed a solution, and are now monitoring the results.
- resolved Feb 16, 2019, 12:22 AM UTC
This incident has been resolved.
- postmortem Feb 25, 2019, 11:44 PM UTC
# Summary Due to an issue during a database migration on 2019-02-14, Drip Campaign activations failed to activate during a 27 hour period from 2019-02-14 12:50 PST to 2019-02-15 15:36 PST. Any Drip Campaign emails scheduled to send before the incident and any deactivations made with the API or automation triggers continued to work as normal. We apologize for any disruption this may have caused for our customers and their business. We have already notified all account administrators for effected accounts. ## What happened Our infrastructure team executed a planned data migration of the database responsible for Drip Campaigns. During this time all new Drip Campaign activations were to be made redundant across both the old and new datasets. An error in our code to configure this redundancy caused loss of data during this migration period. This ultimately prevented any new Drip Campaigns from properly activating. ## What was affected During the incident window: - Drip Campaign stopped activating via the API - Drip Campaign stopped activating via automation triggers - No new Drip Campaigns could be created ## Who was affected Any accounts that activated a Drip Campaign during the period above would have been affected. This includes Drip Campaign activations through the API and through Automation Triggers. If this was the case, your account administrator An email has been sent to your Sendwithus administrators if your account was affected. ## What Sendwithus is doing to fix it While do mistakes happen, it's most important to us that they're caught early so they can be swiftly corrected. To mitigate the likelihood of this happening in the future we will be: - Adjusting our alert thresholds for this particular issue. - Adding more thorough migration testing processes