Lucidworks experienced a minor incident on April 11, 2023 affecting Lucidworks Platform and Connected Search, lasting 7d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 11, 2023, 04:45 PM UTC
Some Connected Search requests to the Query API are intermittently timing out, which results in them returning either empty or inconsistent results. This issue is affecting all Connected Search customers, as well as the Experience Optimizer section of the Springboard UI (platform.lucidworks.com). Lucidworks engineers are investigating the cause of this issue and updates will provided here as soon as we have them.
- investigating Apr 11, 2023, 06:04 PM UTC
We have identified the source of intermittent query failures and occasional timeouts, and are working with an upstream vendor to resolve them. We do not have an ETA for full resolution at this time, but believe that our initial troubleshooting has already lessened both the frequency and severity of these query issues. We will continue to provide updates as the situation develops.
- identified Apr 11, 2023, 08:26 PM UTC
We continue to work to fully resolve the source of the query timeouts, while minimizing the impact as much as possible. We believe we have identified the root cause and are now pursuing corrective actions. We'll continue to provide updates as soon as more information is available.
- monitoring Apr 11, 2023, 09:14 PM UTC
We have confirmed the underlying cause of this issue, and it has been mitigated at this time. A particularly resource-intensive query was found to be impacting the Connected Search backend for all customers by triggering a bug in the underlying data store service. Disabling this query has returned service to normal, and all Connected Search and Springboard UI functionality should be fully restored at this time. Lucidworks will follow up on a root-cause analysis with the upstream vendor in question, and will update our own findings here as we have more details to share.
- monitoring Apr 13, 2023, 10:49 PM UTC
Our upstream vendor has identified the cause of the bug and is working on an urgent fix. In the meantime, we are keeping our mitigation in place and continuing to monitor to ensure continued operations and functionality of Connected Search for all customers. We will provide further updates as soon as we have them.
- resolved Apr 18, 2023, 03:57 PM UTC
Our upstream vendor has deployed a fix for the bug that was impacting some Connected Search customers, and Lucidworks has verified that the issue is now fully resolved.
- postmortem Apr 18, 2023, 03:57 PM UTC
Customers affected by this incident may request a detailed Event Analysis Report via their TAM or through Lucidworks Support.