Kustomer incident
AI Agent messages via WhatsApp and SMS are delayed - Prod 1 and 2
Kustomer experienced a minor incident on September 5, 2025 affecting Channel - SMS and Channel - WhatsApp, lasting 4h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 05, 2025, 10:31 PM UTC
Kustomer is aware of an event affecting AI Agent messages via WhatsApp and SMS that may cause outbound messages to be delayed. Our team has identified the cause and is currently working toward implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.
- monitoring Sep 05, 2025, 10:58 PM UTC
Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- monitoring Sep 05, 2025, 11:28 PM UTC
Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- monitoring Sep 06, 2025, 12:09 AM UTC
Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is currently working to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- monitoring Sep 06, 2025, 01:00 AM UTC
Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is continuing to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- monitoring Sep 06, 2025, 01:40 AM UTC
Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is continuing to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- monitoring Sep 06, 2025, 02:13 AM UTC
Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- monitoring Sep 06, 2025, 02:46 AM UTC
Kustomer has implemented an update to address an event affecting AI Agent messages via WhatsApp and SMS that caused outbound messages to be delayed. Our team is continuing to redrive the affected messages. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- resolved Sep 06, 2025, 03:05 AM UTC
Kustomer has verified the successful redrive of all messages on the WhatsApp channel.
- postmortem Sep 24, 2025, 01:56 PM UTC
# **Summary** On September 5, 2025, some outbound AI responses messages were delayed or failed to be sent. This primarily affected the WhatsApp channel. A code change introduced late on September 4, 2025, caused certain messages to be rejected by our processing system. This led to messages being stuck in a pending state until engineering teams identified and fixed the issue. # **Root Cause** Engineering identified the change that introduced the issue and implemented fixes to ensure messages could be properly processed. Once the fixes were deployed, all affected messages were retried and successfully delivered. # **Timeline** **Sep 4, 2025** * 10:00 pm EST: A system update was deployed. This unintentionally caused some AI and WhatsApp messages to enter a stuck state. **Sep 5, 2025** * Morning EST: Alerts indicated delayed message delivery. Engineers began investigating the issue. * 5:16 pm EST: Fixes were deployed to restore message delivery. Pending outbound messages began processing successfully. * 10:56 pm EST: All pending outbound messages were verified as sent. All systems fully operational. # **Lessons/Improvements** * \[IN PROGRESS\] We are enhancing monitoring and alerting to detect message delivery issues faster, so our teams can respond more quickly in the future. * \[IN PROGRESS\] We are expanding our testing approach to better simulate real-world message flows across all channels. This will help us catch downstream issues earlier in the development process before they reach production.