Kustomer experienced a minor incident on December 21, 2024 affecting Search, lasting 54m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 21, 2024, 10:07 AM UTC
Kustomer is aware of an event affecting Searches in instances hosted in our PROD2 environment that may cause delays in Searches refreshing. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at [email protected] for any further questions or updates.
- identified Dec 21, 2024, 10:17 AM UTC
Kustomer is aware of an event affecting Searches in instances hosted in our PROD2 environment that may cause delays in Searches refreshing. Our team has identified the cause of this issue and is actively working to implement a full resolution. Please expect additional updates within the next 30 minutes and reach out to Kustomer Support at [email protected] for any further questions or updates.
- identified Dec 21, 2024, 10:46 AM UTC
Kustomer continues to work on the event affecting Searches in instances hosted in our PROD2 environment that may cause delays in Searches refreshing. We are actively working to address the issue to restore full functionality. Further updates will follow as progress is made. Please expect additional updates within the next 30 minutes and contact Kustomer Support at [email protected] if you have any further queries.
- monitoring Dec 21, 2024, 10:56 AM UTC
Kustomer has implemented an update to address an event on Searches in instances hosted in our PROD2 environment that were causing delays in Searches refreshing. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at [email protected] if you have additional questions or concerns.
- resolved Dec 21, 2024, 11:02 AM UTC
Kustomer has resolved the event affecting Searches in instances hosted in our PROD2 environment that caused delays in Searches refreshing. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer Support at [email protected] if you have additional questions or concerns.
- postmortem Jan 08, 2025, 09:30 AM UTC
# **Summary** Customers in the Prod2 pod experienced Searches only reflecting new or updated data every 5-10 minutes instead of in real time. # **Root Cause** A configuration change was applied to the pipeline for sending data to search, but a step was missed during the change. As a result, a subset of data encountered errors in the pipeline and was delayed in reaching search. # **Timeline \(EST\)** **Dec 20, 2024** 8:23 PM EDT - A configuration change is applied to the pipeline for sending data to search. **Dec 21, 2024** 3:37 AM EST - The Engineering team is alerted to the issue. 5:14 AM EST - The issue is identified and remediated. The data pipeline is unblocked and the data begins to update correctly. # **Lessons/Improvements** * **Improve internal monitoring for failures to reach search** - The area of the pipeline that encountered the errors did not have monitors that would have picked up on the delayed data. * **Improve internal documentation around the pipeline configuration** - Internal documentation is being updated to document the process around pipeline changes of this nature so this will not happen again.