Kustomer experienced a minor incident on May 8, 2025 affecting Channel - WhatsApp, lasting 2h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 08, 2025, 08:09 PM UTC
Kustomer is aware of an event affecting our WhatsApp Integration that may cause latency in sending messages. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.
- identified May 08, 2025, 08:58 PM UTC
Kustomer has identified an event in our WhatsApp Integrations that may cause delays in messaging. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.
- monitoring May 08, 2025, 09:36 PM UTC
Kustomer has implemented an update to address an event affecting WhatsApp Channel that caused long delays in sending WhatsApp messages and multiples of the same message being sent. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Email or Chat if you have additional questions or concerns.
- resolved May 08, 2025, 10:21 PM UTC
Kustomer has resolved an event affecting WhatsApp Channel that caused long delays in messages to be delivered. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Email or Chat if you have additional questions or concerns.
- postmortem May 20, 2025, 03:13 PM UTC
# **Summary** Between April 24 and May 8, our system experienced three instances of delayed WhatsApp message processing caused by sudden increases in send volume. While scaling eventually resolved these issues, we have taken steps to prevent future occurrences. These include increased scaling for high-volume events like Mother's Day and the implementation of rate limiting. # **Root Cause** The root cause was an overwhelming surge in message volume, leading to the send queue growing faster than our processing capacity. The autoscaling configuration's response time was insufficient to prevent client impact. To address this, we implemented per-user rate limiting to control queue ingestion and increased baseline resource availability during the Mother's Day period. # **Timeline** **April 24, 2025 - 11:00 AM ET -** Slowdown of sending for WhatsApp messages. **April 24, 2025 - 4:00 PM ET -** Scaling allowed the system to recover. **May 7, 2025 - 12:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 7, 2025 - 3:00 PM ET -** Additional scaling allowed our system to recover. **May 8, 2025 - 2:00 PM ET -** Slowdown of sending for WhatsApp messages. **May 8, 2025 - 5:00 PM ET -** Refined the auto scaling configuration to allow the system to recover. **May 9, 2025 - 3:00 PM ET -** Rate limiting imposed on sending of WhatsApp messages. **May 9, 2025 - 3:30 PM ET -** Increased scaling limit temporarily for holiday weekend # **Lessons/Improvements** * **Rate Limiting** - In order to mitigate the impact of several sends coming at the same time, we imposed rate limiting on WhatsApp sends. Initial implementation was 400 sends per user per minute, with further tuning to come. * **Scaling** - Services connected to the sending of WhatsApp messages were scaled up temporarily for the Mother's Day holiday weekend.