Kustomer Outage History

Started Sep 09, 2026, 07:25 PM UTC· resolved Sep 09, 2026, 07:52 PM UTC

1. investigating Sep 09, 2026, 07:25 PM UTC

   Kustomer has identified an event affecting Kustomer Voice across all PRODS that may cause issues when attempting to make outbound calls. Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes.

2. monitoring Sep 09, 2026, 07:45 PM UTC

   Kustomer has implemented an update to address an event affecting Kustomer Voice across all PRODs. Our team is currently monitoring this update to ensure the issue is fully resolved. In order to fully resolve the issue on your end please have all agents refresh their Kustomer Instance.

3. resolved Sep 09, 2026, 07:52 PM UTC

   Kustomer has resolved an event affecting Kustomer Voice across all PRODs. To resolve this issue, our team has rolled back an update. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. postmortem Oct 02, 2026, 05:55 PM UTC

   ## Summary On September 9, 2026, a service event prevented users from initiating some outbound Voice calls. When affected users selected **Call**, the call did not begin. ## Impact The event affected outbound Voice call initiation for four organizations. Other Voice workflows may have continued to operate normally. No data loss or security impact has been identified. ## Timeline * **September 9, 12:27 PM ET:** A Voice application update was released. * **September 9, approximately 2:00 PM ET:** Reports of affected outbound calls began. * **September 9, approximately 3:19 PM ET:** Mitigation began. * **September 9, approximately 3:30 PM ET:** Outbound Voice calling was restored. ## Root Cause A Voice application update caused some outbound call attempts to fail. The affected update was rolled back. ## Resolution The affected update was rolled back, restoring outbound Voice calling for affected users. The incident investigation is complete. ## Preventative Actions * Improve recovery procedures to reduce service restoration time during future service events. * Add validation coverage for supported outbound-calling configurations before future releases. * Improve monitoring, internal alerting, and user feedback when a call cannot be initiated.

Read the full incident report →

Started Sep 08, 2026, 01:23 PM UTC· resolved Sep 08, 2026, 01:39 PM UTC

1. investigating Sep 08, 2026, 01:23 PM UTC

   Kustomer is aware of an event affecting links and content utilising the default kustomer.help domain (not custom client domains) — including Help Center URLs and CSAT surveys — that may cause visibility issues. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. investigating Sep 08, 2026, 01:24 PM UTC

   We are continuing to investigate this issue.

3. resolved Sep 08, 2026, 01:39 PM UTC

   Kustomer has resolved an event affecting links and content utilising the default kustomer.help domain (not custom client domains) that caused visibility issues. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

4. postmortem Sep 10, 2026, 09:05 PM UTC

   # **Summary** On September 7–8, 2026, some customers were unable to access Knowledge Base pages and related survey links hosted on the default `*.kustomer.help` domain. Customers using the default Knowledge Base domain in the prod1 environment encountered browser certificate warnings that prevented access to Knowledge Base content. Related links using the same domain, including some survey links, were also affected. Customers using custom Knowledge Base domains and shared Knowledge Base domains in other production environments were not affected. # **Root Cause** A security certificate used by the shared Knowledge Base domain expired before its renewal validation was completed. This caused browsers to reject connections to links served through that domain, while the underlying Knowledge Base service remained available. # **Timeline** **Sep 7, 2026** **8:00 PM ET**: Kustomer’s default Knowledge Base Domain certificate expired. At this time, customers may have begun encountering certificate warnings when accessing Knowledge Base content. ‌ **Sep 8, 2026** **8:37 AM ET** Kustomer receives reports of certificate warnings for default Knowledge Base Domain content and related links **8:38 AM ET** Kustomer Engineering is notified and begins our On-Call Incident Management process **9:00 AM ET** Kustomer Engineering identifies the shared-domain certificate issue and begins remediation steps **9:24 AM ET** Kustomer Engineering completes remediation steps **9:37 AM ET** Kustomer’s certificate provider acknowledges the remediation steps were complete and provisions a new certificate for the default Knowledge Base domain, restoring service # **Lessons/Improvements** Kustomer Engineering has a dedicated Infrastructure team that continuously works to maintain and improve Kustomer’s availability, reliability, fault tolerance. As part of our continual investment in these areas, we have identified the following action items related to this issue: * Move the shared-domain certificate to an automated renewal approach. * Add proactive monitoring and alerting for certificate expiration across shared customer-facing domains. * Review other certificates using the same renewal approach and prioritize any needed updates. * Document recovery procedures and ownership for certificate renewal activities.

Read the full incident report →

Started Aug 25, 2026, 01:51 PM UTC· resolved Aug 25, 2026, 02:17 PM UTC

1. monitoring Aug 25, 2026, 01:51 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors (PubNub) affecting real-time messaging for orgs in Prod1 that may cause chat messages, agent notifications, and live updates to be delayed or not appear in real time within the platform. Our team is actively monitoring the incident, and working with the vendor where possible to resolve the issue. You can also follow PubNub's status here: https://status.pubnub.com/incidents/cz6q37z3s1d5. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. resolved Aug 25, 2026, 02:17 PM UTC

   The PubNub-related event previously reported affecting real-time messaging for orgs in Prod1 has been resolved. PubNub has confirmed the issue has been mitigated on their end - no messages were missed and no customer impact occurred during the event. All affected areas remain fully operational. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

Read the full incident report →

Started Jul 30, 2026, 05:43 PM UTC· resolved Jul 30, 2026, 07:15 PM UTC

1. investigating Jul 30, 2026, 05:43 PM UTC

   Kustomer is aware of an event affecting text editor that may cause issues when typing and adding content such as Shortcuts. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via [email protected] for any further questions or updates.

2. monitoring Jul 30, 2026, 06:15 PM UTC

   Kustomer has implemented an update to address an event affecting text editor that may cause issues when typing and adding content such as Shortcuts. Refreshing your browser will fully resolve the issue. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. resolved Jul 30, 2026, 07:15 PM UTC

   Kustomer has resolved an event affecting text editor that may cause issues when typing and adding content such as Shortcuts. To resolve this issue, our team has rolled back to a previous version. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. postmortem Aug 08, 2026, 04:49 AM UTC

   # **Summary** On July 30, 2026, the draft text editor on Kustomer’s Timeline product experienced degraded functionality in certain user workflows. Impact included sporadic cursor behavior, text erasure, and issues with copying and pasting text and shortcuts. # **Root Cause** This issue was introduced by a change to the editor that was not fully caught before release. Due to misalignment in our QA process our pre-release validation did not adequately cover the real-world editing patterns affected by this change. Furthermore, this misalignment contributed to a delay in Kustomer’s understanding of the incident's resolution status. # **Timeline** ## **Jul 30, 2026** **11:12 AM ET:** The editor change was deployed **12:27 PM ET:** Kustomer Technical Support escalated the incident to Kustomer’s OnCall process, notifying Engineering immediately. **12:34 PM ET:** Kustomer Engineering identified the issue and completed a rollback **12:51 PM ET:** Internal testing incorrectly confirms that the issue has been fully resolved, due to some customers reporting the issue no longer presented itself **1:29 PM ET:** Continued reports of issue are received from customers who did not receive the full resolution rollout **1:34 PM ET:** Discovery made that the rollback did not fully deploy to resolve the issue. Kustomer Engineering makes additional change to ensure resolution **2:01 PM ET:** Corrective change deployed, and fix is confirmed # **Lessons/Improvements** Kustomer Engineering maintains a robust CI/CD process, and a multi-environment release process designed to prevent issues of this nature. As part of our continual investment in these areas, we have identified the following action items: * Ensure that our internal pre-production environments align with our customer-facing experience, including but not limited to the editor experience, so that issues of this nature are caught earlier in development * Complete an audit of our automated CI/CD process and test coverage of major features, removing any gaps that are identified

Read the full incident report →

Started Jul 26, 2026, 01:13 AM UTC· resolved Jul 26, 2026, 04:34 AM UTC

1. identified Jul 26, 2026, 01:13 AM UTC

   Kustomer has identified an event affecting platform events in PROD1 orgs that may cause delays on event-based data. Our team is currently working to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any further questions or updates.

2. identified Jul 26, 2026, 01:16 AM UTC

   We are continuing to work on a fix for this issue.

3. identified Jul 26, 2026, 02:03 AM UTC

   Kustomer continues to work on the issue affecting platform events in PROD1 orgs that may cause delays on event-based data. Our team is actively working to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any further questions or updates.

4. monitoring Jul 26, 2026, 02:34 AM UTC

   Kustomer has implemented an update to address an event affecting platform events in PROD1 orgs that caused delays on event-based data. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

5. resolved Jul 26, 2026, 04:34 AM UTC

   Kustomer has resolved an event affecting platform events in PROD1 orgs that caused delays on event-based data. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. postmortem Jul 30, 2026, 04:17 PM UTC

   ## Summary On July 25, 2026, some customers experienced elevated platform latency affecting messaging, conversation updates, routing, voice, and other customer-service workflows. Customers may have seen messages remain in a sending state, delayed inbound or outbound messages, slower page and API responses, delayed routing or assignment, and intermittent voice-call delays. The incident was caused by an unusually large burst of background data-processing activity. This created a sudden increase in demand on shared platform infrastructure. Automatic scaling added capacity, but it could not absorb the burst quickly enough to prevent degradation in dependent workflows. We restored service by increasing available capacity, reducing the rate of the initiating workload, and carefully processing delayed work while monitoring platform health. ## Impact * **Customer effect:** Intermittent platform latency; delayed inbound and outbound messaging; slower conversation and API updates; delayed routing or assignment; and intermittent voice-call delays * **Duration:** Customer-visible degradation began at approximately 5:22 PM ET. Service initially recovered at approximately 7:31 PM ET, but degradation later recurred. Broad platform performance was restored and the incident was resolved at 10:30 PM ET. * **Scope:** The incident affected a subset of customers within one production environment. We did not identify corresponding customer-facing degradation in other production environments. ## Timeline * **Approximately 5:22 PM ET:** Processing latency and message backlogs began increasing. * **6:51 PM ET:** The incident response team began coordinated investigation and recovery. * **Approximately 7:00–7:23 PM ET:** We identified affected processing paths and began carefully recovering delayed work at controlled rates. * **7:31 PM ET:** Additional capacity had come online, customer-facing performance had materially improved, and the incident was initially resolved while monitoring continued. * **8:28 PM ET:** The incident was reopened after renewed reports of intermittent degradation. * **Approximately 8:38 PM ET:** Investigation confirmed that messaging, routing, channel, and voice symptoms shared the same underlying platform dependency. * **Approximately 9:52 PM ET:** We reduced the rate of the initiating background workload to protect customer-facing traffic. * **10:08 PM ET:** Monitoring confirmed that workload pressure had fallen substantially and platform health continued improving. * **10:30 PM ET:** After continued monitoring confirmed recovery, the incident was resolved. * **After resolution:** Recovery of remaining delayed work continued at controlled rates under monitoring. ## Root cause A burst of high-volume background data-processing activity generated more downstream work than shared platform infrastructure could safely absorb over a short period. Automatic scaling responded and added capacity, but capacity was added incrementally and did not come online quickly enough for the size and speed of the burst. While the platform was catching up, increased request latency and timeouts affected customer-facing workflows that depended on the same infrastructure. ## Resolution We restored platform performance by: * Increasing available capacity and improving the rate at which additional capacity could be added. * Reducing the rate of the initiating background workload. * Recovering delayed work at controlled rates to avoid creating another traffic spike. * Continuing monitoring after customer-facing performance returned to expected levels. ## Preventative actions We are taking the following actions to reduce the likelihood and impact of recurrence: * Add stronger limits and backpressure controls for high-volume background operations. * Improve the speed at which shared infrastructure scales during sudden traffic increases. * Improve isolation between background processing and latency-sensitive customer workflows. * Expand alerting for rapid backlog growth, resource pressure, and unusual workload patterns. * Strengthen controlled recovery procedures for delayed work. * Expand burst-load and failure-recovery testing for shared platform services. ## Current status Platform performance returned to expected levels, and the initiating workload remained controlled. We continued monitoring service health and the recovery of delayed work after resolution.

Read the full incident report →

Started Jun 25, 2026, 10:24 PM UTC· resolved Jun 25, 2026, 11:30 PM UTC

1. investigating Jun 25, 2026, 10:24 PM UTC

   Kustomer is aware of an event affecting PROD 1 that may affect the ability to send outbound messages for MessageBird. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. identified Jun 25, 2026, 10:49 PM UTC

   Kustomer has identified the cause affecting PROD 1 that is impacting the ability to send outbound messages through MessageBird. Please expect another update within the next 30 minutes, as we reach a resolution. If you have any additional questions or concerns, please reply to this conversation or contact Kustomer Support at [email protected]. Thank you for your patience while we work to resolve this issue.

3. identified Jun 25, 2026, 11:05 PM UTC

   Kustomer has identified and implemented a fix for the issue affecting PROD 1 that impacted the ability to send outbound messages through MessageBird. Our team is actively monitoring the platform to ensure the fix remains effective and that service has fully recovered. If you continue to experience issues sending outbound MessageBird messages, please contact Kustomer Support at [email protected]. We will provide another update once monitoring is complete or if there are any significant developments. Thank you for your patience.

4. monitoring Jun 25, 2026, 11:06 PM UTC

   Kustomer has identified and implemented a fix for the issue affecting PROD 1 that impacted the ability to send outbound messages through MessageBird. Our team is actively monitoring the platform to ensure the fix remains effective and that service has fully recovered. If you continue to experience issues sending outbound MessageBird messages, please contact Kustomer Support at [email protected]. We will provide another update once monitoring is complete or if there are any significant developments. Thank you for your patience.

5. resolved Jun 25, 2026, 11:30 PM UTC

   Kustomer has resolved the issue affecting PROD 1 that impacted the ability to send outbound messages through MessageBird. Our team has verified that the fix has been successfully deployed and service has been restored. If you continue to experience issues sending outbound MessageBird messages, please contact Kustomer Support at [email protected] We apologize for the disruption and appreciate your patience while we worked to resolve the issue.

6. postmortem Aug 06, 2026, 02:13 PM UTC

   ## **Summary** On June 25, 2026, some customers were unable to send outbound WhatsApp replies from within Kustomer. The issue affected reply sending for a subset of WhatsApp channel configurations, which disrupted agent workflows and prevented some automated outbound messages from being sent through the same path. The issue was identified and resolved the same day. Service was fully restored, and the platform is operating normally. ## **Root cause** A change released earlier that day introduced stricter validation in the outbound WhatsApp reply flow. For a subset of supported channel configurations, valid sender values were incorrectly rejected before messages were sent. This caused reply attempts in those configurations to fail. The issue was limited to specific WhatsApp channel setups and did not affect all WhatsApp traffic equally. ## **Timeline** * **June 25, 2026, approximately 22:12 UTC** — Reports began coming in that outbound WhatsApp replies were failing for some customers. * **Shortly after detection** — Investigation confirmed the issue was tied to a recently released validation change in the outbound reply flow. * **June 26, 2026, approximately 01:07 UTC** — A fix was deployed and reply sending was restored. ## **Lessons and improvements** * Validation changes for messaging flows now require broader test coverage across supported channel configuration variants before release. * Additional safeguards are being added to reduce the risk of valid outbound requests being rejected. * Monitoring and regression checks around outbound messaging paths are being strengthened to detect similar issues more quickly.

Read the full incident report →

Started Jun 12, 2026, 02:31 PM UTC· resolved Jun 12, 2026, 09:38 PM UTC

1. investigating Jun 12, 2026, 02:31 PM UTC

   Kustomer is aware of an ongoing WhatsApp issue that may cause outbound messages to not be delivered to end users. While the issue appears to originate with WhatsApp, our team is actively monitoring the situation and working closely to assess the impact on the platform. We will continue to provide updates as more information becomes available. Please expect an additional update within the next 30 minutes. If you have any questions, please contact Kustomer Support at [email protected].

2. identified Jun 12, 2026, 02:46 PM UTC

   Kustomer has identified an ongoing issue with WhatsApp that may cause outbound messages to not be delivered to end users. Our team is closely monitoring the situation and actively working to assess impact. Clients can refer to https://metastatus.com/whatsapp-business-api for the latest WhatsApp service status. Please expect further updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

3. identified Jun 12, 2026, 03:17 PM UTC

   Kustomer continues to monitor the ongoing issue on Whatsapp that may cause outbound Whatsapp messages to not be delivered to end users. Clients can also refer to https://metastatus.com/whatsapp-business-api for the latest WhatsApp service status. Please expect further updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] if you have additional questions or concerns

4. identified Jun 12, 2026, 04:35 PM UTC

   Kustomer continues to monitor the ongoing issue on Whatsapp that may cause outbound Whatsapp messages to not be delivered to end users. Clients can also refer to https://metastatus.com/whatsapp-business-api for the latest WhatsApp service status. Please expect further updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] if you have additional questions or concerns

5. monitoring Jun 12, 2026, 07:29 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting ALL PRODS that may cause Issues with WhatsApp messages not being delivered to end users. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. We are working on redriving previously unsent messages to ensure all messages are sent successfully. Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. resolved Jun 12, 2026, 09:38 PM UTC

   WhatsApp has confirmed that all services are fully recovered. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. postmortem Jun 22, 2026, 02:57 PM UTC

   ## Summary On June 12, 2026, customers using WhatsApp through Kustomer experienced a service event that prevented some outbound messages from being delivered to end users. During the event, messages could appear as sent in Kustomer before final delivery was confirmed by WhatsApp. Service recovered the same day, and affected messages from the incident window were successfully reprocessed. ## Root cause This event was caused by a disruption in WhatsApp Business Platform services operated by Meta. While that disruption was active, Kustomer was unable to complete delivery for some outbound WhatsApp messages and queued affected messages for retry after the upstream service recovered. ## Timeline * **10:14 AM EDT** — Customer reports led to investigation of outbound messaging behavior. * **10:34 AM EDT** — Meta reported high disruptions affecting WhatsApp Business Platform services. * **1:45 PM EDT** — Reprocessing of queued messages began as upstream recovery progressed. * **2:32 PM EDT** — Meta reported recovery for the remaining affected WhatsApp services. * **4:07 PM EDT** — Active message sending was healthy again. * **4:34 PM EDT** — Messages from the incident window had been successfully reprocessed. ## Lessons and improvements * We are improving monitoring and alerting for queued WhatsApp messages so degraded delivery is identified earlier. * We are documenting a safer reprocessing procedure for queued WhatsApp traffic to reduce the chance of retry-related rate limiting during recovery. * We are reviewing backlog-handling procedures to make recovery more predictable when an upstream provider disruption occurs. At this time, system health for WhatsApp message delivery through Kustomer is stable.

Read the full incident report →

Started Jun 09, 2026, 02:00 PM UTC· resolved Jun 09, 2026, 03:18 PM UTC

1. investigating Jun 09, 2026, 02:00 PM UTC

   Kustomer is aware of an event affecting Chat and other channels that may cause delayed message delivery and slower conversation load times. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes and please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. monitoring Jun 09, 2026, 02:11 PM UTC

   Kustomer has identified the cause of the event affecting Chat and other channels that resulted in delayed message delivery and slower conversation load times. The situation has stabilized and our team is continuing to monitor to ensure full resolution. Please reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

3. resolved Jun 09, 2026, 03:18 PM UTC

   Kustomer has resolved an event affecting Chat and other channels that caused delayed message delivery and slower conversation load times. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

4. postmortem Jun 11, 2026, 09:18 PM UTC

   **Summary** On June 9, 2026, customers experienced delays in chat message delivery and related real-time updates for a portion of the morning. The issue was caused by a third-party service event affecting the infrastructure used to propagate real-time messaging updates. Service recovered the same morning, and current system health is stable. **Root cause** A third-party real-time messaging provider experienced elevated latency and intermittent publish failures in one of its regions. That disruption delayed delivery acknowledgements and other real-time updates in Kustomer. Messages continued to be stored successfully, but some updates were not reflected immediately in the client until the external service recovered or the client refreshed. **Timeline** * **9:05 AM EDT** — Customer impact began, including delayed chat message delivery and stale real-time updates. * **9:45 AM EDT** — The third-party provider reported active latency affecting its service. * **9:48 AM EDT** — The provider reported mitigation and recovery. * **Shortly after recovery** — Real-time behavior returned to normal and monitoring confirmed stability. **Lesson/improvements** * We added additional monitoring tied to the third-party provider's public service health signals so similar service events can be identified faster. * We are improving alerting around real-time messaging errors to reduce time to diagnosis. * We are continuing to review operational signals to better distinguish external service issues from internal platform issues.

Read the full incident report →

Started Jun 02, 2026, 02:47 PM UTC· resolved Jun 02, 2026, 03:53 PM UTC

1. investigating Jun 02, 2026, 02:47 PM UTC

   Kustomer is aware of an event affecting chats, emails, and routing that may cause latency with sending, receiving, and routing messages. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.

2. identified Jun 02, 2026, 03:19 PM UTC

   Kustomer has identified an event affecting chats, emails, and routing that may cause latency with sending, receiving, and routing messages. Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. monitoring Jun 02, 2026, 03:27 PM UTC

   Kustomer has implemented an update to address an event affecting chat, email, and routing that caused latency in sending and delivering messages. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. resolved Jun 02, 2026, 03:53 PM UTC

   Kustomer has resolved an event affecting Prod 1 that caused chat, email, and routing latency. To resolve this issue, our team reverted an underlying infrastructure change as a precaution, which helped stabilize the service. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. postmortem Jun 11, 2026, 09:17 PM UTC

   **Summary** On June 2, 2026, customers in our prod1 environment experienced elevated latency affecting chat creation, email sending, and conversation routing. The event caused delayed processing for a subset of customer interactions during the incident window. Service performance recovered the same day after mitigation steps were applied, and the environment is currently operating normally. **Impact** * Scope: prod1 only * Customer effect: delayed chat creation, delayed email sends, and slower conversation routing * Duration: customer-visible impact began in the morning ET on June 2 and materially improved after mitigation later that afternoon * Other environments: prod2 and prod4 were not impacted by the customer-facing degradation **Timeline** * Early June 2: We detected elevated processing latency in services supporting chat, email, and routing in prod1. * Midday to afternoon ET: Customer reports confirmed delayed system behavior in prod1. * Afternoon ET: We increased service capacity and adjusted resource limits to reduce backlog and restore throughput. * Later that day: We completed an infrastructure rollback on affected worker hosts and confirmed stable recovery. * June 3: Monitoring confirmed the environment remained healthy. **Root cause** The incident was caused by infrastructure-level resource exhaustion on a set of worker hosts in prod1. That reduced the availability of a metadata-dependent service and created message backlog, which in turn increased latency for customer-facing workflows such as chat creation, email sending, and routing. The issue was isolated to prod1. **Resolution** We restored service by increasing available capacity, raising resource limits for the affected service, and rolling impacted worker infrastructure back to a stable configuration. After those changes were applied, backlog cleared and latency returned to expected levels. Current system health is stable. **Preventative actions** * Strengthen host-level capacity and disk safeguards before future infrastructure rollouts * Add earlier alerting for infrastructure resource pressure to reduce time to detection * Expand validation for production-scale logging and resource usage prior to promotion * Continue tuning service capacity thresholds and recovery procedures for metadata-dependent workloads

Read the full incident report →

Started May 26, 2026, 10:14 PM UTC· resolved May 27, 2026, 12:21 AM UTC

1. monitoring May 26, 2026, 10:14 PM UTC

   Kustomer has implemented an update to address an event affecting WhatsApp that caused messaging failures. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. resolved May 27, 2026, 12:21 AM UTC

   This incident has been resolved. Our teams have confirmed recovery and platform stability following mitigation efforts. We will be reaching out independently to affected orgs to provide additional context and follow-up as needed. Thank you for your patience while we worked through this issue.

3. postmortem Jun 11, 2026, 09:19 PM UTC

   ## Summary On May 26, 2026, some customers experienced failures when sending WhatsApp messages from existing conversations. The issue affected outbound message delivery for a subset of WhatsApp configurations. Service was restored after the responsible change was rolled back, and the issue is now resolved. ## Impact During the incident window, outbound WhatsApp messages failed for some existing conversations. Newly created conversations were less consistently affected, and impact varied by account configuration. Customers may have seen message-send failures or provider errors while attempting to send WhatsApp messages. ## Timeline All times EDT. * **4:05 PM:** The incident window is believed to have begun after a recent WhatsApp-related change. * **5:54 PM:** Active investigation began. * **5:56 PM:** The team identified a likely connection to recent WhatsApp sender-selection behavior and started a rollback. * **6:05 PM:** Rollback completed. * **6:17 PM – 7:47 PM:** The team validated recovery across affected examples and narrowed the underlying cause. * **8:21 PM:** The incident was declared resolved. ## Root cause The incident was caused by an issue in WhatsApp phone-number lookup and sender selection. A recent change expanded support for multiple valid phone-number formats in certain countries. In some cases, that allowed the system to resolve the wrong sender configuration for an existing conversation. When that happened, outbound sends could target an invalid or unavailable sender identity, causing WhatsApp message delivery to fail. ## Resolution We mitigated the incident by rolling back the relevant WhatsApp change. After the rollback, we verified recovery against affected examples and confirmed that the incident was resolved. ## Preventative actions * Move WhatsApp sender resolution toward more stable identifier-based matching rather than relying on ambiguous phone-number formatting. * Expand regression coverage for country-specific phone-number formatting edge cases. * Improve monitoring and alerting for message delivery failures and queue growth. * Improve testing workflows for WhatsApp changes before production rollout.

Read the full incident report →

Started May 14, 2026, 04:24 PM UTC· resolved May 14, 2026, 05:23 PM UTC

1. investigating May 14, 2026, 04:24 PM UTC

   Kustomer is aware of an event affecting Chats that may cause messages to fail and assistants to not follow the configured flow. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support for any further questions or updates.

2. monitoring May 14, 2026, 04:48 PM UTC

   Kustomer has implemented an update to address an event affecting Chats on Prod 1 that caused messages to fail and assistants to not follow the configured flow. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support if you have additional questions or concerns.

3. resolved May 14, 2026, 05:23 PM UTC

   Kustomer has resolved an event affecting Chat on Prod 1 that caused messages to fail and assistants to not follow the configured flow. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support if you have additional questions or concerns.

4. postmortem Aug 04, 2026, 09:09 PM UTC

   ## **Summary** Between May 13 and May 14, 2026, Kustomer Chat experienced a service event that affected chat availability and performance for a subset of customers. During this period, some customers may have seen intermittent failures, elevated error rates, or degraded behavior in chat-related settings and runtime flows. The event was mitigated through service rollback, additional capacity, and targeted configuration changes. Service health returned to normal after these actions were completed. ## **Root cause** The event was caused by a combination of reduced service capacity during a deployment rollback and higher-than-expected traffic through chat-related request paths. Under those conditions, the affected chat service became unstable and restarted repeatedly, which reduced available capacity further and increased customer-facing errors. Our investigation also identified specific high-volume request patterns that increased memory pressure during the event. We addressed those patterns with caching, traffic protections, and capacity changes. ## **Timeline** * May 13, 2026, early afternoon ET: We detected elevated instability in the chat service and began incident response. * May 13, 2026, afternoon ET: We rolled back affected changes, increased infrastructure capacity, and stabilized dependent services. * May 13, 2026, evening ET: We continued monitoring after initial mitigation and investigated recurring memory pressure. * May 14, 2026: We deployed additional mitigations, including higher minimum service capacity and request-path protections. * Following the mitigation deployments, service health returned to normal and remained stable. ## **Lessons and improvements** We completed several improvements to reduce the likelihood of recurrence: * Increased minimum service capacity to provide more headroom during deployments and recovery. * Added caching for high-volume chat settings requests. * Hardened URL processing behavior with stricter filtering, failure caching, and concurrency limits. * Added improved memory telemetry to speed up detection and diagnosis of similar issues. * Continued follow-up work on deployment recovery procedures and service safeguards for cross-service rollbacks. ## **Current status** The mitigations above have been deployed, and the affected chat service is operating normally.

Read the full incident report →

Started May 01, 2026, 01:39 PM UTC· resolved May 01, 2026, 02:17 PM UTC

1. identified May 01, 2026, 01:39 PM UTC

   Kustomer is aware of an event affecting Gmail Connectivity and Send & Receive that may cause emails to not route into your Kustomer Platform and impact your ability to send messages on conversations. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat for any further questions or updates.

2. identified May 01, 2026, 01:39 PM UTC

   Kustomer is aware of an event affecting Gmail Connectivity and Send & Receive that may cause emails to not route into your Kustomer Platform and impact your ability to send messages on conversations. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat for any further questions or updates.

3. monitoring May 01, 2026, 02:02 PM UTC

   Kustomer has released a fix for the event affecting Gmail Connectivity and Send & Receive functionality that was causing emails to not forward as expected into your Kustomer Platform and impacted sending functionality on conversations. Our team is currently monitoring the released fix to ensure the issue is resolved. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat for any further questions or updates.

4. resolved May 01, 2026, 02:17 PM UTC

   Kustomer has resolved an event affecting Gmail Connectivity and Send & Receive functionality in Prod 1 and Prod 2. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. postmortem May 05, 2026, 02:06 PM UTC

   ## Summary On May 1, 2026, customers using the Gmail integration experienced a service event that caused Gmail connections to disappear and the email channel to stop functioning. The issue affected customers across different regions, more specifically: * EU-based clients, from 2:40 AM ET. * US-based clients, from 5 AM ET. The issue was fully resolved by 9:20 AM ET. No customer data was lost during this event. Messages that were delayed during the incident were fully recovered once service was restored. Current system health is stable. ## Root cause A configuration issue in the infrastructure used by the Gmail integration prevented replacement service tasks from starting correctly during routine task rotation. As running capacity declined over several hours, the Gmail integration became unavailable. ## Timeline | Time \(ET\) | Event | | --- | --- | | May 1, ~12:30 AM | Service task replacement failures began in production. | | May 1, ~2:40 AM | Engineering was alerted after service capacity dropped to zero in one production environment. | | May 1, 2:40–9:20 AM | Teams investigated the failure, identified the configuration problem, prepared a fix, and deployed it. | | May 1, 9:20 AM | Fix deployed and service restored. | ## Lessons and improvements * We are auditing related infrastructure configurations to identify and correct similar patterns in other services. * We are standardizing how service roles are managed so this class of configuration issue is less likely to recur. * We are improving alerting so teams are notified earlier when running service capacity drops below expected levels, before a full service interruption occurs. * We are adding additional checks to catch configuration drift and invalid service role references earlier in the deployment lifecycle.

Read the full incident report →

Started Apr 24, 2026, 08:03 PM UTC· resolved Apr 24, 2026, 09:47 PM UTC

1. monitoring Apr 24, 2026, 08:03 PM UTC

   Kustomer has identified an event affecting Prod1 that may cause internal API errors and latency. Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. monitoring Apr 24, 2026, 08:32 PM UTC

   Kustomer has identified an event affecting Prod1 that may cause internal API errors and latency. Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. monitoring Apr 24, 2026, 09:02 PM UTC

   Kustomer has identified an event affecting Prod1 that may cause internal API errors and latency. Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. monitoring Apr 24, 2026, 09:31 PM UTC

   Kustomer has implemented an update to address an event affecting Prod1 that caused internal API errors and latency. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. resolved Apr 24, 2026, 09:47 PM UTC

   Kustomer has resolved an event affecting Prod1 that caused API errors and latency. To resolve this issue, our team has released an update. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. postmortem Apr 30, 2026, 06:21 PM UTC

   ## Summary On April 24, 2026, customers in our prod1 environment experienced a service event that caused elevated errors and latency in messaging-related workflows. The broad cross-customer impact was limited to approximately 41 minutes, from 3:26 PM ET to 4:07 PM ET. During that window, some customers saw failed or delayed messaging operations. The immediate platform impact was resolved the same day, and overall system health returned to normal. We then completed follow-up mitigation to stop the underlying event source and reduce the risk of recurrence. ## Impact * Customers in prod1 experienced elevated API errors and latency in messaging-related workflows. * The broad cross-customer impact lasted about 41 minutes. * A subset of messaging workflows failed or were delayed during that period. ## Timeline * **~3:25 PM ET** — A newly enabled automation began processing a large backlog of historical conversations for one tenant. * **3:26 PM ET** — Elevated errors and latency began affecting shared messaging workflows in prod1. * **4:01 PM ET** — We published a status update for the production issue. * **4:07 PM ET** — Broad cross-customer impact ended as the affected services stabilized. * **~5:17 PM ET** — We disabled the triggering automation configuration for the affected tenant. * **~5:23 PM ET** — The remaining retry activity stopped. ## Root cause The event was triggered when a newly enabled automation for one tenant processed a much larger set of eligible conversations than intended. That sudden volume overloaded a shared downstream service and caused elevated errors and timeouts in dependent workflows. The incident was amplified by missing safeguards in how this automation handled backlog volume and retries. In particular, the system did not sufficiently limit the number of conversations processed at once or prevent the same failed work from being retried too aggressively. ## Resolution We restored platform stability during the incident by allowing the affected services to recover under increased capacity, then disabled the triggering automation configuration and cleared the remaining retry backlog. System health is currently normal. ## Preventative actions We are treating the following preventative actions as a priority bug effort. These actions are expected to be resolved by the end of May in accordance with our SLOs: * Prevent newly enabled automation settings from processing large historical backlogs unintentionally. * Add stronger batch limits and tenant-level throttling for this workflow. * Reduce retry amplification by improving how failed work is tracked and re-queued. * Improve error handling so rate-limit conditions are classified correctly and handled with the right retry behavior.

Read the full incident report →

Started Apr 14, 2026, 02:44 PM UTC· resolved Apr 14, 2026, 03:01 PM UTC

1. investigating Apr 14, 2026, 02:44 PM UTC

   Kustomer is aware of an event affecting Knowledge bases and forms that may be displaying a 500 error code and preventing access to these urls. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via kustomer.com for any further questions or updates.

2. resolved Apr 14, 2026, 03:01 PM UTC

   Kustomer has resolved an event affecting Knowledge Bases and forms that caused a 500 error and preventing access to these pages. To resolve this issue, our team has performed a roll back in this area. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support via kustomer.com if you have additional questions or concerns.

3. postmortem Apr 24, 2026, 05:14 PM UTC

   ## **Summary** ‌ On April 14, 2026, customers experienced errors accessing the Kustomer Knowledge Base, with all KB pages returning 500 errors. The issue was caused by a dependency upgrade in a recent deployment that introduced a TLS certificate mismatch in internal service communication. Customer impact began at 9:51 AM ET when the deployment reached the first production environment, and was fully resolved by 10:38 AM ET — a ~47 minute impact window. Engineers identified the root cause and completed a full rollback across all production environments within 12 minutes of the initial alert. ‌ ## **Root Cause** ‌ A recent deployment to the KB service included an upgrade to an internal library that contained a known issue with TLS hostname verification. This caused internal service-to-service requests to fail, resulting in 500 errors for all KB requests. The affected library version had previously been identified as problematic in a staging environment, but the fix had not been fully applied across all services before this deployment reached production. ## **Timeline** ‌ **Apr 14, 2026** ‌ 9:51 AM — Deployment reached the first production environment; customers began experiencing 500 errors when accessing the Knowledge Base ‌ 10:26 AM — Automated alerting fired; incident response began ‌ 10:31 AM — Engineers identified a recent deployment as the likely cause and began investigating rollback options ‌ 10:35 AM — Root cause confirmed as a problematic internal library version; rollback initiated across all production environments ‌ 10:37 AM — Rollback completed on prod1; KB restored for affected customers ‌ 10:38–10:39 AM — Rollback completed across remaining production environments ‌ 10:45 AM — Full KB functionality confirmed restored for all customers ‌ 12:09 PM — Corrected fix deployed to the Knowledge Base service; remediation completed across all other affected services ‌ ## **Lessons/Improvements** * Implementing stricter controls to prevent pre-release or beta library versions from being deployed to production * Improving the process for tracking and completing cross-service remediation work when an issue is identified in one service, to ensure all affected services are addressed * Enhancing our pre-production validation process to improve detection of this class of issue before it reaches production environments

Read the full incident report →

Started Mar 19, 2026, 02:42 PM UTC· resolved Mar 19, 2026, 06:29 PM UTC

1. investigating Mar 19, 2026, 02:42 PM UTC

   Kustomer is aware of an event affecting AI for Customers and Reps that have resulted in them not responding. Our team is currently working to identify the cause for the issue in order to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via kustomer.com for any further questions or updates.

2. identified Mar 19, 2026, 03:44 PM UTC

   Kustomer has identified an event in AIC/AIR that may cause unresponsive functionality. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Kustomer.com for any further questions or updates.

3. identified Mar 19, 2026, 04:16 PM UTC

   Kustomer has identified an event that is causing unresponsiveness in AIC/AIR. Our team is working directly with our database provider in order to implement a resolution for this event. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Kustomer.com for any further questions or updates.

4. identified Mar 19, 2026, 04:58 PM UTC

   Kustomer has identified an event affecting PROD 1 that may cause failing Kustomer AI-Agent features. Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. identified Mar 19, 2026, 05:34 PM UTC

   Kustomer has identified an event affecting PROD 1 that may result in failures with Kustomer AI Agent features. Our team is still actively working to implement a resolution. Please expect further updates within the next 30 minutes, and feel free to reach out to Kustomer Support at [email protected] if you have any additional questions or concerns.

6. monitoring Mar 19, 2026, 05:52 PM UTC

   Kustomer has implemented an update to address an event affecting Prod 1 that caused failures with Kustomer's AI features. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. resolved Mar 19, 2026, 06:29 PM UTC

   Kustomer has resolved an event affecting Prod 1 that caused failures with Kustomer's AI features. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

8. postmortem Mar 27, 2026, 03:38 PM UTC

   ## Summary On March 19, 2026, some Kustomer AI features were unavailable for customers hosted in one production environment \(Prod1\). The disruption affected AI-powered responses and related AI workflows for 41 customers in our Prod 1 instance. Other production environments were not affected. Service was fully restored within 4 hours, and the platform is operating normally. ## Root cause The incident was caused by a production database index change that was introduced during a service deployment. That change caused database performance to degrade significantly for a high-traffic dataset used by our AI services. As performance deteriorated, dependent services were unable to start or process requests normally, which led to broad disruption across affected AI features. Recovery was prolonged by duplicate records created during the incident window, which complicated restoration of normal database constraints and required additional remediation before services could be brought back cleanly. ## Timeline All times below are in EDT \(UTC-4\) on March 19, 2026. * **~10:27 AM** — A production deployment introduced a database index change that degraded performance for AI-related services. * **~10:37 AM** — Customer impact was confirmed and incident response began. * **~10:42 AM** — We published a public status update and began active mitigation. * **~11:37 AM** — We identified the primary cause and focused recovery on database stability and service restoration. * **~12:10 PM** — After scaling database capacity and reducing load, service recovery began. * **~1:43 PM** — AI services began recovering and queued work started draining. * **~2:19 PM** — Backlogged work had been processed and core functionality was restored. * **Later that afternoon** — Follow-up cleanup was completed and normal safeguards were re-applied. ## Lessons and improvements We take this incident seriously and are making changes to reduce the likelihood of recurrence. * We are tightening how database schema and index changes are deployed in production, including stronger pre-deployment validation and safer rollout sequencing. * We are removing index-management behavior from service startup paths where it can create unnecessary risk during deployment. * We are improving monitoring and alerting so service health failures and database stress are detected earlier. * We are refining incident response procedures, access readiness, and operational runbooks to speed mitigation during future incidents. * We are reviewing capacity and resilience safeguards for this part of the platform to better handle abnormal database load.

Read the full incident report →

Started Feb 20, 2026, 10:01 PM UTC· resolved Feb 21, 2026, 12:01 AM UTC

1. monitoring Feb 20, 2026, 10:01 PM UTC

   Kustomer has implemented an update to address an event affecting inbound Voice calls in PROD 1 & 2 that caused inbound calls to drop before getting answered by an agent. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Email or Chat if you have additional questions or concerns.

2. investigating Feb 20, 2026, 10:21 PM UTC

   Kustomer is aware of an event affecting Voice conversations that may cause inbound calls to not be accepted properly in the system. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

3. investigating Feb 20, 2026, 10:53 PM UTC

   Kustomer is continuing to investigate an event impacting Voice conversations that may cause inbound calls to not be accepted properly within the system. Our engineering team is actively working to identify the root cause and implement a resolution as quickly as possible. We will provide another update within the next 30 minutes or sooner as more information becomes available. If you have any urgent questions or need assistance, please contact Kustomer Support via Chat or Email.

4. identified Feb 20, 2026, 11:08 PM UTC

   Kustomer is aware of an issue affecting inbound call acceptance in our PROD1 environment where inbound calls were not properly being accepted in the system. Our team has identified the cause of the issue within the code and are actively working on a fix to restore system behavior back to normal. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at Email or Chat Channels if you have additional questions or concerns.

5. monitoring Feb 20, 2026, 11:33 PM UTC

   Kustomer has implemented an update to address an event affecting Voice conversations in PROD 1 that caused inbound calls to be dropped when trying to accept calls. Our team is currently monitoring this update to ensure the issue is fully resolved. If your agents are continuing to experience this issue please have them refresh their browser to apply the fix. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Email or Channel if you have additional questions or concerns.

6. resolved Feb 21, 2026, 12:01 AM UTC

   Kustomer has resolved an incident affecting Voice in PROD 1 that caused inbound calls to drop after being initially accepted. To remediate the issue, our team rolled back a recent code change to a previously stable version, which successfully resolved the error. After thorough monitoring, we have confirmed that all impacted services are fully restored. If any agents continue to experience issues, please have them refresh their browser to ensure the fix is applied. If you have any additional questions or concerns, please reach out to Kustomer Support via Email or Chat.

7. postmortem Mar 02, 2026, 04:53 PM UTC

   ## Summary On February 20, 2026, a frontend change intended to improve performance unintentionally disrupted call setup in our Voice experience. As a result, agents could see an incoming call and click accept/decline, but audio would not connect and calls would time out. We rolled back the change and then deployed an additional fix to make startup more reliable and prevent recurrence. ## What happened When the Voice widget starts, it needs to receive a small set of configuration settings \(including feature-flag values\) from the main app before it can route and connect calls correctly. A pre-existing timing edge case meant that, in some cases, that initial configuration could be sent before the Voice widget was fully ready to receive it. Historically, the widget would receive the same configuration again shortly afterward, which masked a state issue. The performance change reduced those repeated configuration sends \(which is normally a good optimization\). But because the widget sometimes missed the first configuration during startup, it could start with incomplete settings and fall back to legacy call-routing behavior. In that fallback mode, calls could appear answerable in the UI but fail to fully connect audio. ## Root cause A performance optimization changed how/when configuration settings were delivered during Voice widget startup. Combined with a pre-existing startup timing edge case, some sessions did not receive the required configuration in time and fell back to legacy call-routing behavior, preventing audio from being bridged correctly. ## Timeline * 3:04 PM ET: Frontend change deployed * 4:38 PM ET: Reports received that calls could not be answered \(accept/decline visible, no audio\) * 4:44 PM ET: Initial rollback attempt of an unrelated change did not resolve the issue * 4:50 PM ET: Incident communication initiated via status page * 6:24 PM ET: Frontend change reverted/rolled back * 8:47 PM ET: Reports continued for some agents * 10:31 PM ET: Identified that affected sessions could persist without a browser refresh due to cached UI state; refresh/new session restored correct behavior * Saturday 9:00 AM ET: Confirmed reports that calls were successfully connecting ## Lessons / improvements * Harden Voice widget startup so required configuration can’t be missed during initialization. * Add automated smoke/E2E coverage for core Voice call flows \(answer/connect audio\) to detect regressions before production. * Improve safeguards and monitoring around call-connect failures to reduce time-to-detect and time-to-mitigate.

Read the full incident report →

Started Feb 12, 2026, 03:11 PM UTC· resolved Feb 12, 2026, 06:07 PM UTC

1. monitoring Feb 12, 2026, 03:11 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting Prod1 that may cause AIC and AIR failures within the platform. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. OpenAI's status page for this incident can be found here: https://status.openai.com/incidents/01KH94NGSXNH9H4WBPXB3RFZWX Please expect further updates within the next 3 hours, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. resolved Feb 12, 2026, 06:07 PM UTC

   OpenAI has confirmed that all services are fully recovered. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

Read the full incident report →

Started Jan 27, 2026, 07:16 PM UTC· resolved Jan 28, 2026, 11:10 PM UTC

1. investigating Jan 27, 2026, 07:16 PM UTC

   Kustomer is aware of an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. investigating Jan 27, 2026, 07:27 PM UTC

   Kustomer is aware of an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

3. investigating Jan 27, 2026, 07:31 PM UTC

   Kustomer is aware of an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. investigating Jan 27, 2026, 07:58 PM UTC

   Kustomer is aware of an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is continuing to work to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

5. investigating Jan 27, 2026, 08:29 PM UTC

   Kustomer is aware of an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is continuing investigate the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. investigating Jan 27, 2026, 09:03 PM UTC

   Kustomer is aware of an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is continuing to work to investigate the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

7. identified Jan 27, 2026, 09:32 PM UTC

   Kustomer has identified an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is currently working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

8. identified Jan 27, 2026, 09:51 PM UTC

   Kustomer has identified an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is continuing to work to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

9. identified Jan 27, 2026, 10:23 PM UTC

   Kustomer has identified an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team continues to work to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

10. identified Jan 27, 2026, 11:01 PM UTC

    Kustomer has identified an event affecting Satisfaction surveys that may cause surveys to not be sent, once conversations are marked done within the platform. Our team is working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

11. monitoring Jan 28, 2026, 12:16 AM UTC

    The team has identified the problem and mitigations have been applied. Jobs are gradually catching up and the team continues to monitor.

12. investigating Jan 28, 2026, 10:05 PM UTC

    Kustomer has resolved an event affecting Satisfaction Surveys that may cause Surveys to not be sent. To resolve this issue, our team has released an update. After careful monitoring, our team has determined that our systems are now fully restored, but our engineering team is still redriving surveys that did not originally send. During this redriving period lingering issues may still be present. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

13. resolved Jan 28, 2026, 11:10 PM UTC

    Kustomer has resolved an event affecting Satisfaction Surveys that may cause Surveys to not be sent. To resolve this issue, our team has released an update. After careful monitoring, our team has determined that our systems are now fully restored, but our engineering team is still redriving surveys that did not originally send. During this redriving period lingering issues may still be present. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

14. postmortem Feb 04, 2026, 09:50 PM UTC

    # Post Mortem: Chat, Voice Routing, CSAT, Oauth, Scheduled Send Issues # **Summary** On January 22, 2024, customers experienced chat and voice conversations failing to route to agents due to a recent change in the assistant service. This triggered cascading failures across multiple backend services, degrading platform performance for several orgs. On January 27th, a subsequent incident occurred due to our scheduled jobs queue being flooded by assistant service jobs. This caused some CSAT Surveys to not send, Oauth connections to fail to refresh, and scheduled messages to not send. ‌ **Root Cause** A recent change to the assistant backend service increased the maximum workflow loops before transferring a conversation to an available agent. This allowed a single rate-limited WhatsApp conversation to become stuck in an infinite retry loop while attempting to transfer. The transfer requests themselves were also rate-limited, overwhelming shared infrastructure \(e.g. “job” engine which is shared by CSAT\) and causing service degradation across multiple orgs. # **Timeline** **Jan 22, 2026** 1:33 PM – Users began experiencing errors with chat and voice conversations failing to route to agents 1:44 PM – Engineers identified the problematic deployment and initiated rollback across all environments 1:54 PM – Rollback completed across all environments; on-call engineers continued monitoring system status 2:02 PM – Full assistant functionality restored for all customers 6:07 PM - Customers begin to report Oauth connections that failed to refresh ‌ **Jan 24, 2026** 6:24 PM - Customers begin to report that scheduled messages were not sending ‌ **Jan 27, 2026** 1:00 PM - Customers begin to report that CSAT surveys are not being sent 4:00 PM - Identified cause of CSAT scheduled job processing delays 6:00 PM - Started a script to manually increase processing throughput of scheduled jobs ‌ **Jan 28, 2026** 12:08 PM - Deployed code change to programmatically increase processing throughput of scheduled jobs 3:49 PM - Backlog of all delayed jobs processed and system restored ‌ **Lessons/Improvements** * Implementing new alerts to detect when scheduled job processing falls behind, enabling faster identification of similar issues * Improving alert prioritization to reduce noise and ensure critical alerts are acted upon immediately * Enhancing monitoring for downstream service dependencies * Evaluating queue architecture changes to prevent a single conversation from impacting other customers \("noisy neighbor" isolation\) * Investigating improvements to make our job scheduling service more resilient to backlogs * Creating documentation of all services that depend on scheduled jobs to better understand incident ripple effects

Read the full incident report →

Started Jan 22, 2026, 06:44 PM UTC· resolved Jan 22, 2026, 07:30 PM UTC

1. investigating Jan 22, 2026, 06:44 PM UTC

   Kustomer is aware of an event affecting Chat and Voice conversations that may cause the conversation to not be routed to an available agent. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Email or Chat for any further questions or updates.

2. monitoring Jan 22, 2026, 07:02 PM UTC

   Kustomer has implemented an update to address an event affecting Chats and Voice calls in PROD 1 & 2 that caused conversations to not be routed to available agents. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at Chat and Email if you have additional questions or concerns.

3. resolved Jan 22, 2026, 07:30 PM UTC

   Kustomer has resolved an event affecting Conversational Assistants in PROD1 and 2 that caused conversations to not route to available agents. To resolve this issue, our team has completed a rollback our codebase to address the failures in the assistants. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at Chat or Email if you have additional questions or concerns.

4. postmortem Feb 04, 2026, 09:52 PM UTC

   # Post Mortem: Chat, Voice Routing, CSAT, Oauth, Scheduled Send Issues # **Summary** On January 22, 2024, customers experienced chat and voice conversations failing to route to agents due to a recent change in the assistant service. This triggered cascading failures across multiple backend services, degrading platform performance for several orgs. On January 27th, a subsequent incident occurred due to our scheduled jobs queue being flooded by assistant service jobs. This caused some CSAT Surveys to not send, Oauth connections to fail to refresh, and scheduled messages to not send. ‌ **Root Cause** A recent change to the assistant backend service increased the maximum workflow loops before transferring a conversation to an available agent. This allowed a single rate-limited WhatsApp conversation to become stuck in an infinite retry loop while attempting to transfer. The transfer requests themselves were also rate-limited, overwhelming shared infrastructure \(e.g. “job” engine which is shared by CSAT\) and causing service degradation across multiple orgs. # **Timeline** **Jan 22, 2026** 1:33 PM – Users began experiencing errors with chat and voice conversations failing to route to agents 1:44 PM – Engineers identified the problematic deployment and initiated rollback across all environments 1:54 PM – Rollback completed across all environments; on-call engineers continued monitoring system status 2:02 PM – Full assistant functionality restored for all customers 6:07 PM - Customers begin to report Oauth connections that failed to refresh ‌ **Jan 24, 2026** 6:24 PM - Customers begin to report that scheduled messages were not sending ‌ **Jan 27, 2026** 1:00 PM - Customers begin to report that CSAT surveys are not being sent 4:00 PM - Identified cause of CSAT scheduled job processing delays 6:00 PM - Started a script to manually increase processing throughput of scheduled jobs ‌ **Jan 28, 2026** 12:08 PM - Deployed code change to programmatically increase processing throughput of scheduled jobs 3:49 PM - Backlog of all delayed jobs processed and system restored ‌ **Lessons/Improvements** * Implementing new alerts to detect when scheduled job processing falls behind, enabling faster identification of similar issues * Improving alert prioritization to reduce noise and ensure critical alerts are acted upon immediately * Enhancing monitoring for downstream service dependencies * Evaluating queue architecture changes to prevent a single conversation from impacting other customers \("noisy neighbor" isolation\) * Investigating improvements to make our job scheduling service more resilient to backlogs * Creating documentation of all services that depend on scheduled jobs to better understand incident ripple effects

Read the full incident report →

Started Dec 10, 2025, 03:38 AM UTC· resolved Dec 10, 2025, 04:52 AM UTC

1. identified Dec 10, 2025, 03:38 AM UTC

   Kustomer is aware of an event affecting outbound webhook delivery in our Prod2 environment. Customers may experience failures due to 404 and 503 errors when the platform attempts to send outbound webhooks. Our team is actively investigating the cause and working toward a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] if you have additional questions or concerns.

2. identified Dec 10, 2025, 03:55 AM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Dec 10, 2025, 04:22 AM UTC

   Kustomer has implemented a fix for the event impacting outbound webhooks in the Prod2 and Prod4 environments. We are backporting this fix across environments, and our team is monitoring to ensure service stability and continued recovery. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

4. resolved Dec 10, 2025, 04:52 AM UTC

   Kustomer has resolved the event that impacted outbound webhooks in the Prod2 and Prod4 environments. The backported fix has been fully deployed, and webhook delivery is functioning as expected across all affected environments. Our team has verified stability, and no further impact is anticipated. If you have additional questions or concerns, please reach out to Kustomer support at [email protected]

5. postmortem Feb 12, 2026, 04:57 PM UTC

   ### **Summary** On December 9, 2025, Kustomer experienced an interruption to outbound webhook delivery affecting customers in our **prod-2 \(EU\)** and **prod-4 \(IN\)** production regions. During this time, some outbound webhook requests were unable to reach customer endpoints, resulting in delivery failures and error responses. The disruption occurred during a routine platform update related to our deployment systems. An underlying inconsistency from a previous infrastructure change caused a required backend service to become temporarily unavailable while updates were in progress. As a result, webhook endpoints could not accept or process delivery attempts for a limited period. Engineering teams identified the issue quickly and restored full service shortly thereafter. Once the affected systems were brought back online, normal webhook delivery resumed and any queued events were successfully processed. There was no permanent data loss, and webhook functionality returned to normal operation across all impacted regions. ### **Impact** * **Duration:** Approximately 1 hour and 40 minutes * **Scope:** Production regions prod-2 and prod-4 * **Customer Impact:** * Outbound webhook delivery failures during the incident window * Webhook requests may have returned HTTP 503 or 404 responses * Webhook events were delayed but not lost ### **Next Steps** While safeguards already exist to protect core platform services, we are implementing additional improvements to reduce the likelihood of similar disruptions in the future. These include strengthening validation during platform updates, improving detection of incomplete infrastructure changes, and enhancing internal visibility when critical services are modified.

Read the full incident report →

Started Dec 02, 2025, 10:44 PM UTC· resolved Dec 03, 2025, 12:49 AM UTC

1. monitoring Dec 02, 2025, 10:44 PM UTC

   Kustomer is aware of an event reported by one of our third party vendors affecting the observability of AI Agent for Reps and AI Agent for Customers that may cause trace logs to not show up in AI Agent logs within conversations. Our team is monitoring the incident, and working with the vendor where possible to resolve the issue. Please expect further updates within the next 3 hours, and reach out to Kustomer support over Chat or Email if you have additional questions or concerns.

2. resolved Dec 03, 2025, 12:49 AM UTC

   A third party incident has been resolved affecting PROD 1 AI conversations (both AIC and AIR) to not include trace logs. Functionality has returned in this area Please reach out to Kustomer support through Chat or Email if you have additional questions or concerns.

Read the full incident report →

Started Nov 13, 2025, 08:56 PM UTC· resolved Nov 14, 2025, 04:01 AM UTC

1. investigating Nov 13, 2025, 08:56 PM UTC

   Kustomer is aware of an event affecting kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

2. investigating Nov 13, 2025, 09:31 PM UTC

   Kustomer is aware of an event affecting kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. Our team is continuing to work toward identifying the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support [email protected] for any further questions or updates.

3. investigating Nov 13, 2025, 10:10 PM UTC

   Kustomer is aware of an issue impacting emails sent from @kustomerapp.com addresses, which may result in delivery failures to Gmail recipients. Our team is actively investigating the cause and working to implement a resolution. We will provide another update within 30 minutes. For further questions or updates, please contact Kustomer Support via Chat or Email.

4. investigating Nov 13, 2025, 10:43 PM UTC

   Kustomer continues to investigate an issue affecting emails sent from @kustomerapp.com addresses, which may cause delivery failures to Gmail (@gmail.com) recipients. Our team is working diligently to identify the root cause and implement a fix. We will share another update within 30 minutes. For any questions or further assistance, please contact Kustomer Support via Chat or Email.

5. investigating Nov 13, 2025, 11:27 PM UTC

   Kustomer is still investigating an ongoing issue impacting emails sent from @kustomerapp.com addresses, which may result in delivery issues when reaching Gmail (@gmail.com) recipients. Our team continues to work toward identifying the root cause and implementing a resolution. We will provide another update within 30 minutes. For any questions or additional support, please reach out to Kustomer Support via Chat or Email.

6. identified Nov 13, 2025, 11:45 PM UTC

   Kustomer has identified an event affecting mail.kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

7. identified Nov 14, 2025, 12:17 AM UTC

   Kustomer has identified an event affecting mail.kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

8. identified Nov 14, 2025, 12:49 AM UTC

   Kustomer has identified an event affecting mail.kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

9. identified Nov 14, 2025, 01:20 AM UTC

   Kustomer has identified an event affecting mail.kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

10. identified Nov 14, 2025, 01:59 AM UTC

    Kustomer has identified an event affecting mail.kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. Our team is still continuing to work on implementing a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

11. identified Nov 14, 2025, 02:34 AM UTC

    Kustomer has identified an event affecting mail.kustomerapp.com email addresses that may cause messages sent to gmail.com addresses to bounce. A fix is being rolled out incrementally to customers and we continue to monitor the situation as it improves. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

12. identified Nov 14, 2025, 03:04 AM UTC

    A fix is being rolled out incrementally to customers and we continue to monitor the situation as it improves. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via Chat or Email for any further questions or updates.

13. monitoring Nov 14, 2025, 03:17 AM UTC

    Kustomer has implemented a fix, which we are currently monitoring, to address the event affecting [POSTMARK] ISP errors. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support via Chat or Email if you have additional questions or concerns.

14. resolved Nov 14, 2025, 04:01 AM UTC

    Issues encountered during sending emails to gmail addresses have been resolved. Within the hour, the issue should be fully resolved as changes propagate throughout DNS records. If you experience any further problems, please reach out to support and we'll look into it with high priority.

15. postmortem Dec 04, 2025, 02:25 PM UTC

    # **Summary** New, tightened bulk sender rules by Gmail went into effect in November 2025, which affected Kustomer’s ability to deliver transactional emails to Gmail mailboxes. Specifically, DMARC alignment failures due to messages not being aligned to the from domain resulted in emails sent from [mail.kustomerapp.com](http://mail.kustomerapp.com) domain to [gmail.com](http://gmail.com) addresses to be blocked. ‌ **Root Cause** Beginning in early November 2025, Gmail rolled out stricter requirements for organizations that send large volumes of email. Because Kustomer sends more than 5,000 messages per day to Gmail addresses, Gmail classifies our traffic as “bulk sender” traffic and applies stronger authentication checks. Many of the emails sent from Kustomer customers through Postmark were not fully meeting Gmail’s updated authentication requirements. Although the messages were being sent from addresses that looked like they belonged to each customer \(eg: [[email protected]_](mailto:[email protected])\), the DMARC policy check was now failing. As a result, Gmail could not verify that these messages truly came from the domain shown in the “From” address. When Gmail applied its new enforcement rules, any email that did not clearly authenticate as coming from the correct domain was blocked before delivery. This caused a spike in rejected emails for customers sending to Gmail inboxes. ‌ # **Timeline** **Early November 2025** Gmail begins enforcing newly updated bulk-sender requirements for high-volume senders. These rules require stricter domain authentication for all messages sent to Gmail inboxes. **Nov 11, 2025** ‌ **2:18 PM EST — Initial Reports**Customer support teams begin receiving reports that emails sent to Gmail addresses are bouncing or failing to deliver. **2:46 PM EST — Issue Confirmed** Responding teams identify that the issue is widespread and related to Gmail’s new enforcement behavior. **3:11 PM EST — Root Cause Narrowed Down**Misalignment in required email authentication settings is identified as the likely cause of Gmail rejecting messages. **4:00 PM EST - 7:45 PM EST – Fix Validation** Responding engineers work on implementing and validating the fix. **8:01 PM EST — Fix deployment**The necessary configuration updates are confirmed, including correcting DNS records so Gmail can properly verify the sender. **10:00 PM–11:00 PM EST — Remediation & Stabilization**Updated DNS records are applied across affected customer domains. Gmail begins accepting previously rejected messages, and email delivery to Gmail addresses returns to normal. ‌ **Lessons/Improvements** **Improved Email Authentication Setup Across All Customers:** All customer organizations now have the correct DNS records and Postmark sender signatures in place to ensure their messages meet Gmail’s authentication requirements. **Enhanced Monitoring for Email Delivery Issues:** We are implementing new logging and alerting to detect spikes in email bounces earlier, allowing us to respond more quickly if delivery issues arise in the future. **Stronger Setup Process for New Integrations:** Going forward, all new Postmark email integrations will be configured with the correct authentication settings from the start to prevent this issue from occurring.

Read the full incident report →

Started Nov 08, 2025, 07:06 AM UTC· resolved Nov 08, 2025, 08:44 AM UTC

1. investigating Nov 08, 2025, 07:06 AM UTC

   Kustomer is aware of an event affecting our Knowledge Base rendering it inaccessible. [PROD 2] Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.

2. investigating Nov 08, 2025, 07:39 AM UTC

   Our Engineering team is still actively investigating the issue alongside our cloud provider to identify the underlying cause and implement a resolution. We’ll provide an update within the next 30 minutes.

3. investigating Nov 08, 2025, 08:11 AM UTC

   We are continuing to investigate the issue and working with our cloud provider to identify the cause in an effort to implement a resolution. You can expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates in the meantime.

4. monitoring Nov 08, 2025, 08:19 AM UTC

   A fix has been deployed and services have been restored. We’re closely monitoring system performance to confirm stability.

5. resolved Nov 08, 2025, 08:44 AM UTC

   Kustomer has resolved the event affecting PROD 2 that caused the Knowledge Base to be inaccessible. To resolve this issue, our team has deployed a fix. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

6. postmortem Nov 14, 2025, 06:38 PM UTC

   ## **Summary** ‌ During a routine deployment, a configuration issue caused an SSL certificate mismatch that temporarily affected access to some knowledge bases in one production region. ## **Root Cause** ‌ The incident was caused by a configuration change that introduced a new SSL certificate missing the required Subject Alternative Names \(SANs\) for two production regions. These SANs are necessary for secure connections between CloudFront and the load balancers serving knowledge base traffic. Without the correct entries, HTTPS requests to those regions failed TLS validation, resulting in users being unable to access their knowledge bases. ‌ The underlying issue stemmed from how the certificate was selected and applied during the deployment process. The new certificate shared the same primary domain as others in production but lacked the additional SANs specific to these regions. This mismatch caused CloudFront to reject requests at the TLS handshake stage, effectively disrupting service availability for knowledge bases in both affected regions. ## **Timeline** ‌ **Nov 7, 2025** **8:48 PM EST** Release went live to all regions **9:37 PM EST** Alarm triggered for low invocations in our Prod2 region for KB indexing **Nov 8, 2025** **12:33 AM EST** Incident triggered by on-call engineer **2:45 AM EST** Certificates identified as the likely culprit of this incident **3:15 AM EST** Fix applied and system fully recovered ## **Lessons/Improvements** * **Reduce noisy alarms** - Multiple noisy alarms of the same type at the same time for normally functioning systems caused us to miss this alarm. * **Status**: In progress. We are actively tuning our alarms to reduce noise and ensure actual alarms are actionable. * **Standardize certificates across all regions** - Standardize our certificates for all regions to ensure that we do not have custom certificate configurations which could cause incidents in select regions. * **Status**: In progress. A fix has been applied, but further work to standardize is ongoing. * **More KB logging** - Our knowledge base front-end has limited logging in place to help us identify issues faster. **Status**: Planned. Implement new logging like we have in our other front-end platforms to ensure faster alerting.

Read the full incident report →

Started Nov 05, 2025, 08:39 AM UTC· resolved Nov 05, 2025, 09:19 AM UTC

1. investigating Nov 05, 2025, 08:39 AM UTC

   Kustomer is aware of an event currently impacting tag functionality, which may result in tag search and usage not working as expected. Searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also be temporarily impacted. Our Engineering team is actively investigating the issue to determine the underlying cause and implement a resolution. We’ll provide an update within the next 30 minutes. We sincerely apologize for the disruption and appreciate your patience and understanding. For any additional questions or updates, please don’t hesitate to contact Kustomer Support at [email protected].

2. identified Nov 05, 2025, 09:12 AM UTC

   Kustomer has identified an event currently impacting tag functionality, which may result in tag search and usage not working as expected. Searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also be temporarily impacted. Our Engineering team has identified the cause and is implementing mitigations to restore full functionality. We’ll continue to provide updates every 30 minutes until the issue is fully resolved. For any additional questions or updates, please don’t hesitate to contact Kustomer Support at [email protected].

3. identified Nov 05, 2025, 09:12 AM UTC

   Kustomer has identified an event currently impacting tag functionality, which may result in tag search and usage not working as expected. Searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also be temporarily impacted. Our Engineering team has identified the cause and is implementing mitigations to restore full functionality. We’ll continue to provide updates every 30 minutes until the issue is fully resolved. For any additional questions or updates, please don’t hesitate to contact Kustomer Support at [email protected].

4. resolved Nov 05, 2025, 09:19 AM UTC

   Kustomer has resolved an event affecting tag functionality and tag search, which caused tag usage and search to not work as expected. During the incident, searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also have been temporarily impacted. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support at [email protected] if you have any additional questions or concerns.

5. postmortem Nov 17, 2025, 04:09 PM UTC

   ## **Summary** A minor refactor to the search service introduced an unexpected dependency issue that caused several platform-level search features, including tag and shortcut searches, to fail temporarily. ## **Root Cause** The problem stemmed from a minor refactor that changed the order of internal operations within the search service. As a result, the system attempted to retrieve cluster information before the appropriate index was initialized. This caused requests for specific search contexts—such as tags, shortcuts, and snippets—to be routed incorrectly, leading to errors in those API calls. The impact was limited to searches using tag selection \(for example, in timelines, bulk actions, editing business rules, and editing workflows\), snippet and shortcut searches, and the Team Pulse “Skills” page. Normal customer searches, reporting, and automations were unaffected. The issue was not detected in pre-production testing because the testing environment’s configuration differed from production, masking the dependency sequencing problem. ## **Timeline** **Nov 4, 2025** **4:30 PM EST** Release went live to all regions **Nov 5, 2025** **3:20 AM EST** Incident triggered after receiving reports from customers **4:00 AM EST** Identified the suspect release **4:06 AM EST** Rolled back the suspect release **4:15 AM EST** All search functionality fully recovered ## **Lessons/Improvements** * **Implement new tests** - We need to set up automated test coverage that covers these specific endpoints that errored. * **Status**: Planned. Implement new automated tests covering these search cases to catch issues automatically faster. * **Add documentation to code** - The code that was changed is legacy code with minimal documentation with historical context. * **Status**: Planned. Adding new documentation to the affected code to ensure future safer changes.

Read the full incident report →

Started Oct 21, 2025, 01:57 AM UTC· resolved Oct 21, 2025, 06:33 PM UTC

1. identified Oct 21, 2025, 01:57 AM UTC

   Kustomer has identified an event in AI Agent Observability (AIC and AIR) on Prod1 that may prevent new observability traces from ingesting into the platform. Existing traces remain available. Our team is working to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any questions or concerns.

2. identified Oct 21, 2025, 02:44 AM UTC

   Kustomer has observed an issue affecting AI Agent Observability (AIC and AIR) on Prod1, resulting in new observability traces not being ingested into the platform. Existing traces remain accessible. Our team continues to work toward a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at [email protected] for any questions or concerns.

3. identified Oct 21, 2025, 03:08 AM UTC

   Kustomer is aware of delays affecting AIC and AIR metrics ingestion on Prod1 due to an ongoing AWS OpenSearch Serverless service disruption. We’re working with our cloud provider toward recovery, which is currently expected within the next two hours. We continue to monitor the situation and will share additional updates as more information becomes available. For any questions or concerns, please contact [email protected].

4. identified Oct 21, 2025, 05:02 AM UTC

   Kustomer is aware of delays affecting AIC and AIR metrics ingestion on Prod1 due to an ongoing AWS OpenSearch Serverless service disruption. We’re working with our cloud provider toward recovery. We continue to monitor the situation and will share additional updates as more information becomes available. For any questions or concerns, please contact [email protected].

5. identified Oct 21, 2025, 06:37 AM UTC

   We’re continuing to work with our cloud provider to resolve the delays affecting AIC and AIR metrics ingestion on Prod1, related to the ongoing AWS OpenSearch Serverless service disruption. Our Engineering teams are closely monitoring recovery progress and we will continue to share updates as more information becomes available. If you have any questions or concerns, please contact Kustomer Support at [email protected].

6. identified Oct 21, 2025, 10:52 AM UTC

   Kustomer continues to work with our cloud provider to address the delays on AIC and AIR metrics ingestion on Prod1, related to the ongoing AWS OpenSearch Serverless service disruption. Our Engineering teams are monitoring recovery progress, and we’ll continue to share further updates as new information becomes available. If you have any questions or concerns, please contact Kustomer Support at [email protected].

7. identified Oct 21, 2025, 03:42 PM UTC

   Kustomer continues to work with our cloud provider to address the delays on AIC and AIR metrics ingestion on Prod1, related to the ongoing AWS OpenSearch Serverless service disruption. Our Engineering teams are monitoring recovery progress. We are experiencing incremental recovery, and we should continue to see recovery as time passes. We’ll continue to share further updates as new information becomes available. If you have any questions or concerns, please contact Kustomer Support at [email protected].

8. monitoring Oct 21, 2025, 05:39 PM UTC

   Kustomer has worked with our cloud provider to address the delays on AIC and AIR metrics ingestion on Prod1, related to the ongoing AWS OpenSearch Serverless service disruption. Our Engineering teams are continuing to monitor recovery progress, and we’ll continue to share further updates as new information becomes available. If you have any questions or concerns, please contact Kustomer Support at [email protected].

9. resolved Oct 21, 2025, 06:33 PM UTC

   Kustomer has resolved an event affecting AI Agent Observability (AIC and AIR) on Prod1 that may prevent new observability traces from ingesting into the platform. To resolve this issue, our team has worked with our cloud provider to address the delays. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at [email protected] if you have additional questions or concerns.

10. postmortem Oct 29, 2025, 05:27 PM UTC

    # **Summary** On October 20, 2025, Kustomer experienced a significant service disruption affecting all customers on our platform. The incident was triggered by a major AWS service disruption in the us-east-1 region that impacted ~140 AWS services, including core infrastructure components our platform depends on. The incident occurred in two distinct phases: * Phase 1 \(3:00 AM - 6:30 AM EDT\): Initial AWS service disruption causing widespread connectivity issues and elevated error rates across our platform. Services began recovering as AWS addressed the underlying DNS resolution issues. * Phase 2 \(9:20 AM - 5:00 PM EDT\): Secondary wave of issues, including a critical authentication problem that prevented users from logging into the platform from 11:47 AM to 4:30 PM EDT. This simultaneously overlaps with the disruption to AWS EC2 service on October 20 2:48am EDT to 4:50pm EDT. This authentication issue was caused by our disaster recovery automation during our attempts to establish failover capabilities while the AWS EC2 disruption was ongoing. Full platform recovery was achieved by 5:00 PM EDT on October 20, with final cleanup operations completed by 7:00 PM EDT. A remaining issue with AIC and AIR observability infrastructure was fully resolved by 11:45 AM EDT on October 21. **Root Cause** **Primary Cause: AWS Regional Service Disruption** The incident originated from a widespread AWS service disruption in the us-east-1 region. According to AWS, the root cause was a DNS resolution failure for internal service endpoints, specifically affecting DynamoDB regional endpoints. This DNS issue cascaded into AWS's EC2 internal launch system, creating widespread connectivity problems across the region. **Secondary Cause: Disaster Recovery Automation Issue** During our disaster recovery response, an infrastructure configuration issue emerged that prevented proper authentication for approximately 4.5 hours. When disaster recovery preparations were initiated at 11:25 AM EDT, our infrastructure automation discovered that primary and secondary region configurations shared the same management context. As the automation began provisioning resources in the secondary region, it simultaneously triggered scaling changes in the primary region's authentication service, reducing it to a minimal operational state. Under normal circumstances, an automated deployment would have immediately corrected this configuration. However, this deployment failed due to the ongoing AWS service disruptions, leaving the authentication service's load balancer in an inconsistent state—pointing to a configuration with insufficient capacity while healthy instances remained unreachable. This was an edge case that required the specific combination of: \(1\) initiating disaster recovery operations, \(2\) the infrastructure management coupling between regions that were partially operational \(us-east-1 was significantly degraded but remained operational\), and \(3\) simultaneous AWS service failures in us-east-1 preventing the standard recovery mechanism from completing. Had us-east-1 been completely offline, there would not have been extra consideration required for operations in that region. # **Timeline** _All times in Eastern Daylight Time_ ### **Initial Service Disruption Phase** **Oct 20, 2025** **3:11 AM** - Multiple alerts triggered indicating elevated error rates across platform services. Kustomer’s Statuspage was accessible but oncall responders could not authenticate successfully to provide updates. **3:18 AM** - Incident response initiated; engineers begin investigation **4:25 AM** - Confirmed correlation with AWS service disruptions in us-east-1 region **5:01 AM** - AWS identifies root cause of DNS resolution failures **5:27 AM - 6:03 AM** - AWS reports significant recovery; Kustomer platform functionality observed to be restored. Access to Kustomer’s Statuspage was restored at 5:37am. **6:30 AM** - Initial recovery verified; platform communications channels tested successfully. Kustomer platform is verified fully functional ### **Secondary Service Disruption Phase** **8:55 AM** - New wave of elevated error rates detected across multiple services **9:42 AM** - AWS announces mitigation in progress with EC2 throttling. Additional compute is significantly limited and affects autoscaling operations. **10:11 AM** - Severity escalated; separate incident channel created **10:40 AM** - Decision made to initiate disaster recovery preparations in us-east-2 region as contingency **11:02 AM** - Engineers begin provisioning infrastructure in secondary region **11:46 AM -** Due to authentication issues, Kustomer Technical Support began responding to client inquiries directly through email to continue correspondence **11:47 AM** - **3:48 PM** Multiple customer reports of authentication failures; disaster recovery efforts are taking place during this time. At 3:48pm, the root cause for authentication errors: load balancer configuration prematurely routing traffic to secondary region **4:03 PM** - AWS reports ongoing recovery across most services **4:15 PM** - Load balancer configuration corrected **4:30 PM** - Customer authentication restored; users able to access platform **5:00 PM** - API traffic normalized; error rates return to baseline **5:48 PM** - AWS announces full recovery of regional services in us-east-1 **6:01 PM - 6:35 PM** - Queues redriven to recover delayed operations **7:00 PM** - Secondary region infrastructure fully prepared for future failover needs ### **Extended Recovery** **8:40 PM** - Observability logs for AIC and AIR feature remain impacted due to Opensearch Serverless recovery mitigations implemented by AWS **Oct 21, 2025** **11:45 AM** - Full recovery of Opensearch Serverless which restores AIC and AIR observability ‌ **Lessons/Improvements** ### **What Went Well** **Disaster recovery readiness**: Our recent disaster recovery exercise was most recently conducted in July 2025. The updated documentation proved valuable during the incident. The team was able to reference established runbooks and procedures, even as we encountered unexpected challenges. **Cross-team coordination**: Engineers across multiple teams collaborated effectively to simultaneously address recovery in the primary region while preparing failover capabilities in the secondary region. **Iterative improvement**: The team identified and implemented fixes to disaster recovery automation in real-time, improving our processes even during the incident. **Customer communication**: Our status page experienced authentication issues during the incident, limiting our ability to communicate with customers. Additionally, Kustomer’s own access to the platform was impacted from this service disruption. Our tech support and customer success teams were able to keep customers updated by reaching out via the email channel. ### **Areas for Improvement** **Earlier detection of infrastructure anomalies**: The authentication service load balancer inconsistency was observed earlier in the incident but wasn't immediately prioritized for investigation amid numerous other AWS-related issues. We're implementing additional health checks and automated detection to ensure redundant and proactive alerting for this critical functionality as well as updating runbooks to reinforce the importance of investigating the minor anomalies during major incidents. **Status page reliability**: The ability to keep customers updated to the latest status about the Kustomer platform is critical to our operations and maintaining trust. We are exploring options to improve the reliability of this tool. ### **Planned Improvements** **Immediate Actions \(Completed or In Progress\):** * Exploring redundant customer communication channels beyond our primary status page * Separated infrastructure management for secondary region to enable independent deployment and faster disaster recovery * Adding safeguards to disaster recovery automation and race conditions to prevent unintended traffic shifts and validate target health before routing changes **Strategic Initiatives:** * Conducting more frequent disaster recovery exercises to validate our failover procedures under realistic conditions * Developing faster, more reliable automation scripts and orchestration tools for disaster recovery operations to reduce manual intervention and accelerate failover execution * Establishing disaster recovery readiness as a mandatory requirement in our engineering standards, ensuring all new features and services are designed with multi-region capabilities and tested regularly for failover scenarios ### **Commitment to Reliability** This incident reinforced our commitment to building a resilient platform. While we cannot prevent cloud provider service disruptions, we can minimize their impact through better disaster recovery capabilities, more robust automation safeguards, and regular testing of our failover procedures. We recognize that our customers depend on Kustomer for critical business operations, and we take that responsibility seriously. The improvements outlined above represent concrete steps toward faster recovery times and reduced impact from future infrastructure disruptions. We appreciate the patience of our customers during this incident and remain committed to continuous improvement of our platform's reliability and resilience.

Read the full incident report →