[TAGS] Issue with Searching and Using Tags [ALL PRODS]

Update timeline

1. investigating Nov 05, 2025, 08:39 AM UTC

   Kustomer is aware of an event currently impacting tag functionality, which may result in tag search and usage not working as expected. Searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also be temporarily impacted. Our Engineering team is actively investigating the issue to determine the underlying cause and implement a resolution. We’ll provide an update within the next 30 minutes. We sincerely apologize for the disruption and appreciate your patience and understanding. For any additional questions or updates, please don’t hesitate to contact Kustomer Support at [email protected].

2. identified Nov 05, 2025, 09:12 AM UTC

   Kustomer has identified an event currently impacting tag functionality, which may result in tag search and usage not working as expected. Searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also be temporarily impacted. Our Engineering team has identified the cause and is implementing mitigations to restore full functionality. We’ll continue to provide updates every 30 minutes until the issue is fully resolved. For any additional questions or updates, please don’t hesitate to contact Kustomer Support at [email protected].

3. identified Nov 05, 2025, 09:12 AM UTC

   Kustomer has identified an event currently impacting tag functionality, which may result in tag search and usage not working as expected. Searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also be temporarily impacted. Our Engineering team has identified the cause and is implementing mitigations to restore full functionality. We’ll continue to provide updates every 30 minutes until the issue is fully resolved. For any additional questions or updates, please don’t hesitate to contact Kustomer Support at [email protected].

4. resolved Nov 05, 2025, 09:19 AM UTC

   Kustomer has resolved an event affecting tag functionality and tag search, which caused tag usage and search to not work as expected. During the incident, searches involving work items, work sessions, snippets, shortcuts, and the Team Pulse – Skills tab may also have been temporarily impacted. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support at [email protected] if you have any additional questions or concerns.

5. postmortem Nov 17, 2025, 04:09 PM UTC

   ## **Summary** A minor refactor to the search service introduced an unexpected dependency issue that caused several platform-level search features, including tag and shortcut searches, to fail temporarily. ## **Root Cause** The problem stemmed from a minor refactor that changed the order of internal operations within the search service. As a result, the system attempted to retrieve cluster information before the appropriate index was initialized. This caused requests for specific search contexts—such as tags, shortcuts, and snippets—to be routed incorrectly, leading to errors in those API calls. The impact was limited to searches using tag selection \(for example, in timelines, bulk actions, editing business rules, and editing workflows\), snippet and shortcut searches, and the Team Pulse “Skills” page. Normal customer searches, reporting, and automations were unaffected. The issue was not detected in pre-production testing because the testing environment’s configuration differed from production, masking the dependency sequencing problem. ## **Timeline** **Nov 4, 2025** **4:30 PM EST** Release went live to all regions **Nov 5, 2025** **3:20 AM EST** Incident triggered after receiving reports from customers **4:00 AM EST** Identified the suspect release **4:06 AM EST** Rolled back the suspect release **4:15 AM EST** All search functionality fully recovered ## **Lessons/Improvements** * **Implement new tests** - We need to set up automated test coverage that covers these specific endpoints that errored. * **Status**: Planned. Implement new automated tests covering these search cases to catch issues automatically faster. * **Add documentation to code** - The code that was changed is legacy code with minimal documentation with historical context. * **Status**: Planned. Adding new documentation to the affected code to ensure future safer changes.