[TWILIO WHATSAPP & SMS] [Outbbound messages were temporarily not sending] [PROD1]

Update timeline

1. monitoring Mar 06, 2025, 03:22 PM UTC

   Kustomer has implemented an update to address an event affecting Twilio Whatsapp & SMS messages that caused delayed sending. The update has been successfully completed. Any impacted messages that have not sent have will been identified and re-sent. We are still in the process of re-sending and will continue to post updates. Please and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. monitoring Mar 06, 2025, 03:54 PM UTC

   Kustomer has implemented an update to address an event affecting Twilio Whatsapp & SMS messages in Prod 1 that caused delayed sending. The update has been successfully completed. Any impacted messages that have not sent have will been identified and re-sent. We are still in the process of re-sending and will continue to post updates. Please and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. monitoring Mar 06, 2025, 04:23 PM UTC

   Kustomer has implemented an update to address an event affecting Twilio Whatsapp & SMS messages in Prod 1 that caused delayed sending. The update has been successfully completed. Any impacted messages that have not sent have will been identified and re-sent. Our team is still in the process of re-sending any impacted messages and will continue to post updates. Please and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. monitoring Mar 06, 2025, 04:53 PM UTC

   Kustomer has applied an update aimed at resolving an issue in Prod 1 that was delaying the sending of Twilio Whatsapp & SMS messages. This update has been successfully executed. We have identified any messages that were affected and have not yet been sent, and are actively resending them. Our team will continue working on this and will provide further updates as necessary. Should you have any further questions or concerns, please contact Kustomer support at [email protected].

5. monitoring Mar 06, 2025, 05:22 PM UTC

   Kustomer has implemented an update to fix a problem in Prod 1 that was causing delays in the delivery of Twilio Whatsapp & SMS messages. The update has been completed successfully. We have pinpointed messages that were delayed and not yet sent and are currently in the process of resending them. Our team remains committed to addressing this issue and will keep posting updates as needed. If you have any more questions or concerns, please reach out to Kustomer support at [email protected].

6. resolved Mar 06, 2025, 05:42 PM UTC

   Kustomer has addressed an issue impacting Twilio Whatsapp & SMS messages that resulted in delayed transmissions. To rectify this, our team has resent all messages that were not sent during the disruption. Following thorough monitoring, we have confirmed that our systems have been fully restored and all affected messages have been successfully resent. If you have any further questions or concerns, please contact Kustomer support at [email protected].

7. postmortem Mar 10, 2025, 05:47 PM UTC

   ## **Summary** A subset of Twilio SMS and WhatsApp drafts sent from the Kustomer platform between 9:25 AM EST and 9:53 AM EST on March 6th 2025 failed to send and were stuck in a pending state until they were repaired by Kustomer engineering at 12:40PM. ## **Root Cause** Kustomer introduced a change, intended to improve error handling for Twilio SMS and WhatsApp channels. This change introduced a scenario where some drafts would get incorrectly identified as being misconfigured, which prevented them from being sent. This left them in a “Sending” state in the UI until they were manually rescheduled. ## **Timeline** **Mar 6, 2025** **9:20 AM EST** Triggered deployment intended to improve error handling in Kustomer’s Twilio service **9:25 AM EST** Deployment finished **9:41 AM EST** Kustomer was alerted that Twilio SMS and WhatsApp drafts/messages were stuck in “Sending” state, began rolling back change \(as it was the only recent change\) **9:53 AM EST** Rollback is completed, Twilio SMS and WhatsApp drafts/messages are confirmed to be sending again, began monitoring Twilio service **10:00 AM EST** - Started to redrive the Twilio SMS and WhatsApp drafts/messages that were not delivered during the incident window **12:40 PM EST** Finished redriving the Twilio SMS and WhatsApp drafts/messages that were not delivering, all messages sent ## **Lessons/Improvements** * **Automated Tests** - We are adding additional automated tests that test the Twilio service against a broader array of configurations to prevent further regressions of this type. Kustomer is identifying all relevant parameters to ensure thorough test case coverage. **Status**: In Progress * **Runbook for Redriving drafts** - Kustomer will optimize our runbook for redriving undelivered drafts to minimize expected recovery time in future cases of orphaned drafts. **Status**: In Progress * **Additional Review Gates** - Message delivery is of the utmost importance. We have added an additional review step for changes that impact logic that can change the deliverability of a message. **Status**: Complete