[WHATSAPP] latency in message sending and issues with messages persisting [Prod 1]

Update timeline

1. investigating Apr 24, 2025, 06:05 PM UTC

   Kustomer has identified an event in Whatsapp that may cause delays in messages being sent. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

2. investigating Apr 24, 2025, 06:11 PM UTC

   We are continuing to investigate this issue.

3. investigating Apr 24, 2025, 06:12 PM UTC

   We are continuing to investigate this issue.

4. monitoring Apr 24, 2025, 06:39 PM UTC

   Kustomer has implemented an update to address an event affecting latency in WhatsApp affecting Prod 1 that caused latency in messaging. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. monitoring Apr 24, 2025, 07:28 PM UTC

   Kustomer has implemented an update to address an event affecting latencies in WhatsApp messaging that caused messages to be delayed and/or not persist. Our team is continuing to monitor this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. resolved Apr 24, 2025, 08:17 PM UTC

   Kustomer has resolved an event affecting WhatsApp Prod 1 that caused latency in messages being delivered. To resolve this issue, our team has processed a redrive. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. postmortem May 20, 2025, 03:16 PM UTC

   # **Summary** Between April 24 and May 8, our system experienced three instances of delayed WhatsApp message processing caused by sudden increases in send volume. While scaling eventually resolved these issues, we have taken steps to prevent future occurrences. These include increased scaling for high-volume events like Mother's Day and the implementation of rate limiting. # **Root Cause** The root cause was an overwhelming surge in message volume, leading to the send queue growing faster than our processing capacity. The autoscaling configuration's response time was insufficient to prevent client impact. To address this, we implemented per-user rate limiting to control queue ingestion and increased baseline resource availability during the Mother's Day period. # **Timeline** **April 24, 2025 - 11:00 AM ET -** Slowdown of sending for WhatsApp messages. **April 24, 2025 - 4:00 PM ET -** Scaling allowed the system to recover. **May 7, 2025 - 12:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 7, 2025 - 3:00 PM ET -** Additional scaling allowed our system to recover. **May 8, 2025 - 2:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 8, 2025 - 5:00 PM ET -** Refined the auto scaling configuration to allow the system to recover. **May 9, 2025 - 3:00 PM ET -** Rate limiting imposed on sending of WhatsApp messages. **May 9, 2025 - 3:30 PM ET -** Increased scaling limit temporarily for holiday weekend # **Lessons/Improvements** * **Rate Limiting** - In order to mitigate the impact of several sends coming at the same time, we imposed rate limiting on WhatsApp sends. Initial implementation was 400 sends per user per minute, with further tuning to come. * **Scaling** - Services connected to the sending of WhatsApp messages were scaled up temporarily for the Mother's Day holiday weekend.