Higher Logic Thrive Marketing Enterprise (Real Magnet) - Long Sending Delays

Update timeline

1. investigating Oct 05, 2023, 02:20 PM UTC

   We are experiencing long message sending delays for our customers. Our Engineering team is investigating the issue. If you have a message that should've been sent by now, please leave the message as they are and they will be sent out once the delay has been resolved. We apologize for the inconvenience and appreciate your patience.

2. investigating Oct 05, 2023, 03:26 PM UTC

   Our Engineering team continues to investigate this issue. We see that this issue is not only causing sending delays for messages, it is also delaying the following: - Messages sent through campaigns - Deliverability Testing - Send to Individual Tests We'll continue to keep you updated as we learn more.

3. monitoring Oct 05, 2023, 04:42 PM UTC

   Our Engineering team has identified and corrected the issue. Messages are starting to catch up however it may take some time. We'll continue to monitor this afternoon before resolving this.

4. resolved Oct 05, 2023, 08:03 PM UTC

   We have confirmed that this incident is fully resolved. We have not seen further issues in the past few hours. We plan to have a root cause analysis (RCA) ready for distribution in the next 3 business days. Thank you for your patience as we worked to resolve this issue.

5. postmortem Oct 11, 2023, 05:21 PM UTC

   **What Happened** An error occurred processing Real Magnet mailings causing the mailing queue to back up. Email delivery was delayed for about 3 ½ hours. **Timeline \(all times EDT on October 5, 2023\)** * 9:14 AM – Outgoing messages started erroring. * 9:50 AM – First customer reports that mailings are not being sent. * 10:23 AM– Multiple customers are reporting delays in sending their mailings and Application Engineering is notified that a large issue is going to be escalated. Application Engineering also notifies Engineering Development team of the issue. Investigation starts. * 12:36 PM – The cause is identified and resolved. The backlog of mailings begins to be processed. * 1:00 PM – The backlog of mailings was processed. ‌ **Root Cause** A duplicate email was mistakenly added to the Global Suppressed Emails table through Real Magnets internal administration tool. This duplicate email caused an error to be thrown within the logic that processes mailings. **Details** As part of Higher Logic’s monitoring a list of emails that should be suppressed is generated and then manually added to a data table where we assign a primary key. The part of the process that assigns the primary key failed because it returned two of the same email addresses. **Corrective Actions** We will add a unique index on the associated table so that a user will not be able to add an email address that already exists in the table with the same record type. We are working on improving monitoring.