Inline Attachment Issue

Update timeline

1. resolved Jul 14, 2023, 11:07 PM UTC

   A subset of emails sent between 2 am and 11 am PDT July 14th had incorrect inline attachments added to them. Our team has identified and resolved the issue. Sendwithus support will be reaching out to impacted customers.

2. postmortem Jul 19, 2023, 06:20 PM UTC

   # Summary Between 18:33 July 13th and 11:20 July 14th 2023 \(PDT\), inline attachments from a small number of emails were incorrectly attached to an unknown number of emails sent by other accounts. # Impact & Metrics | **Question** | **Answer** | | --- | --- | | What was the impact | Incorrect inline attachments were incorrectly attached to a maximum of 17% of overall emails sent by Sendwithus | | Who was impacted | A subset of Sendwithus users sending emails between July 13th 6:33 pm PDT and July 14th 11:20 am PDT may have been impacted | # Detection & Response‌ | **Question** | **Answer** | | --- | --- | | When was the incident detected | July 14th 2023, 10:21 am PDT | | How was the incident detected | An impacted user reached out to the Sendwithus Customer Support team and the incident was quickly escalated | # Timeline _Times are in Pacific Time_ **Jul 13, 2023** **09:44** A new feature related to sending inline attachments was released **18:33** The first detection of a send using the feature since the change **Jul 14, 2023** **10:21** A user reported some of their recipients were receiving unexpected inline attachments on their emails **10:45** Customer Support notifies on-call engineers and the incident is escalated **11:00** Status page updated **11:10** Resolved the issue by reverting to the previous version of Sendwithus **12:40** Underlying issue identified **13:00** Engineering team began an investigation to determine total user impact **17:00** First emails to personally reach out to impacted users sent # Root Cause Analysis **Problem:** A user has reported that some of their emails contain an unexpected attachment. Why? **Reason:** Recent changes were made to our systems to support sending an arbitrary number of inline attachments. This change introduced a bug which resulted in a variable being reused for subsequent emails sent by the same process. Why? **Reason:** An optional parameter in a function definition had a default reference-type value. The issue was not identified before deploying to production. Why? **Reason:** Our current testing, linting, and code review processes were not sufficient to identify the problem. # What can be improved * Changes are being made to our testing and linting processes to catch this category of bug in our CI pipeline * The transient nature of inline attachments and a lack of thread/process level identification in our archived transaction logs has made it difficult to identify exactly which customers and emails were affected. Due to this only the maximum number of _potential_ impacted emails can be identified * Increased controls to improve cross-tenant isolation of information # Corrective actions * The problematic change was quickly identified and reverted after we became aware of the issue * Changes have since been made to the codebase to reduce the probability of similar issues manifesting elsewhere