Close experienced a minor incident on November 4, 2024 affecting Application UI and Indexing and 1 more component, lasting 45m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 04, 2024, 08:45 AM UTC
We are investigating an issue where our Search Indexer's ingestion of new events is delayed. This may result in the latest changes not being reflected on many pages across the application – Leads Search, Contacts Search, Inbox, Opportunities, Reporting, etc.
- identified Nov 04, 2024, 09:20 AM UTC
We have identified a problem with one of our search databases and have initiated a failover.
- monitoring Nov 04, 2024, 09:29 AM UTC
The failover succeeded. The Indexer is still catching up with the backlog of events. We are monitoring the situation and estimate that everything will be back to normal within 3-5 minutes.
- resolved Nov 04, 2024, 09:31 AM UTC
The issue is resolved now.
- postmortem Nov 07, 2024, 03:06 PM UTC
Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. ## Impact For some clients changes performed in Close took longer than usual to be reflected in our search functionality on November 4th from 8:06 to 9:31 UTC. ## Root Cause and Resolution We are in the process of moving data to a new search infrastructure. Due to encountering a number of unexpectedly large objects during this process the indexing of new changes got delayed. We adjusted the indexing process to truncate the overly large objects to avoid encountering this in the future.