Kustomer incident

[CHAT] [NOT ALLOWING FOR MESSAGES TO BE SENT] [PROD1]

Minor Resolved View vendor source →

Kustomer experienced a minor incident on May 14, 2025 affecting Channel - Chat, lasting 1h 1m. The incident has been resolved; the full update timeline is below.

Started
May 14, 2025, 05:35 PM UTC
Resolved
May 14, 2025, 06:36 PM UTC
Duration
1h 1m
Detected by Pingoru
May 14, 2025, 05:35 PM UTC

Affected components

Channel - Chat

Update timeline

  1. investigating May 14, 2025, 05:35 PM UTC

    Kustomer is aware of an event affecting Chat widgets that may cause Chat messages to not send. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via email at [email protected] for any further questions or updates.

  2. investigating May 14, 2025, 05:40 PM UTC

    Kustomer is aware of an event affecting Chat widgets that may cause Chat messages to not send. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via email at [email protected] for any further questions or updates.

  3. monitoring May 14, 2025, 06:00 PM UTC

    Kustomer has implemented an update to address an event affecting Chat Widgets that caused messages to not be sent in the Chat. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Email '[email protected]' if you have additional questions or concerns.

  4. resolved May 14, 2025, 06:36 PM UTC

    Kustomer has resolved an event affecting Chat messaging that caused Chats to not be allowed to send in the widget. To resolve this issue, our team has rolled back an update to the platform. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Email '[email protected]' if you have additional questions or concerns.

  5. postmortem May 23, 2025, 06:35 PM UTC

    ## **Summary** On May 14, 2025, a change was deployed to the chat widget at 10:56 AM EDT to improve the logout experience. This update aimed to fix an issue where the widget would incorrectly return to a "New Chat" screen after a user logged out \(specifically, calling \`Kustomer.logout\(\)\`\) from a different page, such as a knowledge base article. However, this change inadvertently caused the chat widget to lose access to essential functions needed for sending and receiving messages after a logout and subsequent new chat initiation. This resulted in messages disappearing and the chat window becoming unresponsive. ## **Root Cause** The deployed change, while intended to improve the logout flow, inadvertently caused the application to lose references to critical business logic functions stored within the application state. The application state was incorrectly assumed to only hold data, not these essential functions, leading to their loss during the state restoration process after logout. ## **Timeline** **May 14, 2025** * **10:56 AM EST** Chat widget update deployed. * **1:24 PM EST** Incident declared after reports of issues. * **1:51 PM EST** Error identified related to missing functions in the application state. * **1:53 PM EST** Rollback performed, restoring chat functionality. * **2:54 PM EST** Root cause found and fixed **May 15, 2025** * **10:30 PM EST** All impacted sessions were identified AND cleared from cache ## **Lessons/Improvements** * **Enhanced Testing** - Automated regression tests will be added to prevent future regressions related to the application state. QA testing procedures will be updated to always include testing the ability to start a new conversation after logout. * **Status**: In Progress * **Improved Documentation** - Documentation regarding the expected behavior of the logout function will be enhanced. * **Status**: Done