Smart Search Load Issues (Admin & On-Site)

Update timeline

1. investigating Oct 09, 2024, 09:01 PM UTC

   We’ve identified an issue with Smart Search experiencing intermittent loading problems within both the admin portal and on-site. Our team is fully engaged and actively troubleshooting the problem.

2. identified Oct 09, 2024, 09:15 PM UTC

   We are currently experiencing memory spikes related to the search engines, which we believe are causing the degraded performance. Our team is investigating the issue to resolve it as quickly as possible.

3. monitoring Oct 09, 2024, 09:28 PM UTC

   A fix has been implemented, and search functionality has been fully restored. We are actively monitoring the situation to ensure continued stability.

4. resolved Oct 09, 2024, 09:55 PM UTC

   The team is confident that the issue was resolved as of 17:28 EDT. They are currently conducting a full investigation to determine the root cause. Once identified, we will provide a post-mortem report.

5. postmortem Oct 10, 2024, 01:44 PM UTC

   **Cause:** We observed an unexpected increase in system activity over the past 24-48 hours, which revealed some challenges with the search system's ability to scale effectively under higher demand. **Resolution:** The search team addressed the issue by increasing system resources and rebalancing the search cluster, which restored full production health. **Prevention:** We are collaborating with our partners to conduct a comprehensive review of the incident and will continue to monitor the system to maintain stability. Additionally, there are several potential optimizations that are being evaluated for potential implementation to enhance overall system performance.