[Knowledge Base] [Custom Domains returning 500 error] [POD(s) AFFECTED]

Update timeline

1. investigating Apr 14, 2026, 02:44 PM UTC

   Kustomer is aware of an event affecting Knowledge bases and forms that may be displaying a 500 error code and preventing access to these urls. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via kustomer.com for any further questions or updates.

2. resolved Apr 14, 2026, 03:01 PM UTC

   Kustomer has resolved an event affecting Knowledge Bases and forms that caused a 500 error and preventing access to these pages. To resolve this issue, our team has performed a roll back in this area. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support via kustomer.com if you have additional questions or concerns.

3. postmortem Apr 24, 2026, 05:14 PM UTC

   ## **Summary** ‌ On April 14, 2026, customers experienced errors accessing the Kustomer Knowledge Base, with all KB pages returning 500 errors. The issue was caused by a dependency upgrade in a recent deployment that introduced a TLS certificate mismatch in internal service communication. Customer impact began at 9:51 AM ET when the deployment reached the first production environment, and was fully resolved by 10:38 AM ET — a ~47 minute impact window. Engineers identified the root cause and completed a full rollback across all production environments within 12 minutes of the initial alert. ‌ ## **Root Cause** ‌ A recent deployment to the KB service included an upgrade to an internal library that contained a known issue with TLS hostname verification. This caused internal service-to-service requests to fail, resulting in 500 errors for all KB requests. The affected library version had previously been identified as problematic in a staging environment, but the fix had not been fully applied across all services before this deployment reached production. ## **Timeline** ‌ **Apr 14, 2026** ‌ 9:51 AM — Deployment reached the first production environment; customers began experiencing 500 errors when accessing the Knowledge Base ‌ 10:26 AM — Automated alerting fired; incident response began ‌ 10:31 AM — Engineers identified a recent deployment as the likely cause and began investigating rollback options ‌ 10:35 AM — Root cause confirmed as a problematic internal library version; rollback initiated across all production environments ‌ 10:37 AM — Rollback completed on prod1; KB restored for affected customers ‌ 10:38–10:39 AM — Rollback completed across remaining production environments ‌ 10:45 AM — Full KB functionality confirmed restored for all customers ‌ 12:09 PM — Corrected fix deployed to the Knowledge Base service; remediation completed across all other affected services ‌ ## **Lessons/Improvements** * Implementing stricter controls to prevent pre-release or beta library versions from being deployed to production * Improving the process for tracking and completing cross-service remediation work when an issue is identified in one service, to ensure all affected services are addressed * Enhancing our pre-production validation process to improve detection of this class of issue before it reaches production environments