Lucidworks Outage History

Started Aug 07, 2026, 10:54 PM UTC· resolved Aug 08, 2026, 02:43 AM UTC

1. investigating Aug 07, 2026, 10:54 PM UTC

   A subset of Lucidworks Platform, Lucidworks AI, and Connected Search customers are currently experiencing a service disruption related to invalid credentials. End users relying on search and AI capabilities across impacted clients may experience request failures. The root cause of this issue is currently under investigation, but is initially believed to be related to an unintended secret rotation within the Lucidworks Platform backend that only affects certain customer applications. We are actively investigating the situation to fully determine the cause and mitigate the disruption. We will provide further updates as soon as more information becomes available.

2. monitoring Aug 07, 2026, 11:14 PM UTC

   We have identified the issue to be caused by a background cleanup job that began unintentionally rotating customer application credentials in certain scenarios, beginning around 1700 UTC on August 5. The job runs periodically, and only in specific edge cases will erroneously identify a given set of client credentials as needing to be replaced. We have disabled this job in order to mitigate the issue so that it cannot continue to affect any additional customers. We are producing a report of affected client credentials and Lucidworks Support will be reaching out to each directly via Zendesk to inform them of what needs to be done in order to resolve the issue. A fix for the underlying bug has been prepared and is currently undergoing testing before being rolled out to our production environment. We will post another update here as soon as that has been deployed.

3. resolved Aug 08, 2026, 02:43 AM UTC

   After an exhaustive review of error logs, we have verified that the impact was contained to a handful of Lucidworks Search customers, and have already taken action to rotate all affected credentials. Lucidworks Search implementations include a failsafe mechanism to fall back to lexical-only search in the event that LWAI is unreachable, which would have prevented search errors from any impacted customer environments. Clients with custom Lucidworks Fusion implementations are encouraged to ensure their particular implementation employs a similar mechanism, following our standard best practice guidance. Out of an abundance of caution, we will leave the cleanup job that was at the source of this issue turned off over the weekend. The fix for it has already been tested and verified, and will be deployed to production under scrutiny on Monday, August 10. Any customer still experiencing any issues is encouraged to please follow up with your Lucidworks Customer Excellence representative or Lucidworks Support for further details and assistance.

4. postmortem Aug 18, 2026, 12:18 AM UTC

   ## Summary On August 7, 2026, Lucidworks was alerted to several application credentials being invalidated across Connected Search, Lucidworks AI, and Lucidworks Platform, and declared a Sev1 incident. Lucidworks found that a regular cleanup job had begun erroneously designating active integrations for deletion starting on August 5, 2026, resulting in credentials that were actively in use being incorrectly invalidated and recreated. Lucidworks later determined that this same issue also invalidated embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking. We disabled the cleanup job on August 7 to stop further disruptions, rotated credentials for confirmed impacted customers, and deployed a permanent fix on August 12. No further customer-facing errors have been detected since the fix was deployed. ## Root Cause A recent code change to the backend service that manages customer integrations with a third-party authentication and authorization provider altered how that service checked for outdated integrations across a customer's full set of applications. As a result, the service could only see a partial view of each customer's active integrations rather than the complete list. Due to this, it incorrectly treated valid, active integrations as no longer in use and recreated them, generating new backend credentials in the process. Recreating an integration this way had a secondary effect: it also invalidated the embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking, in addition to the backend Platform credentials. This behavior occurred in irregular, clustered bursts rather than on a steady, predictable schedule, which made the resulting customer impact appear intermittent and difficult to correlate to a single cause. Lucidworks determined the root cause by correlating the timing of the incorrect credential and embed token changes with a recent code change to the affected service, and confirmed the diagnosis by tracing how that service determined which integrations were still active. Lucidworks confirmed the fix by deploying a corrected version of the service and validating that credential and embed token regeneration had stopped. ## Lucidworks Actions Lucidworks has taken and will take the following actions as a result of this incident: * Disabled the automated integration-reconciliation process to immediately stop further credential and embed token disruptions * Deployed a corrected version of the affected service and re-enabled the reconciliation process after validating the fix * Rotated third-party authentication and authorization credentials for confirmed impacted customer environments * We will audit and add consistent customer, application, and request tracking to logs across affected services to reduce future diagnosis time * We will evaluate an underlying third-party library defect identified during this investigation and determine whether an upgrade can remove the need for its current workaround * We will review how integration recreation is handled to reduce the blast radius of future defects, so that a single error cannot invalidate embed tokens for unrelated functionality * We will review our proactive customer outreach process to ensure affected customers are notified consistently, regardless of whether a customer contacts Support directly themselves ## Recommended Client Actions Lucidworks previously recommended that affected customers verify that any embedded Agent Studio widgets, Analytics signal collection Beacons, or usage metrics tracking are functioning normally following this incident, and contact Lucidworks Support if authentication errors persist. Lucidworks communicated this recommendation directly to affected customers prior to publication of this report.

Read the full incident report →

Started Aug 06, 2026, 03:03 PM UTC· resolved Aug 06, 2026, 04:19 PM UTC

1. investigating Aug 06, 2026, 03:03 PM UTC

   Lucidworks is currently responding to a report of service disruption affecting Connected Search functionality. While administrative operations within the Lucidworks Platform dashboard remain fully functional, external search operations for a subset of customers utilizing Connected Search are experiencing impact. End users accessing search through affected external applications are currently unable to receive search results or predictive suggestions. We are actively investigating this issue to identify the underlying cause and restore standard operation. Further updates will be provided as additional details become available.

2. identified Aug 06, 2026, 03:48 PM UTC

   The Connected Search service disruption is limited in scope to a specific client configuration. Affected customers are experiencing 401 Unauthorized response codes related to invalid authorization credentials when attempting to perform search queries through this integration, preventing end users from receiving search results or suggestions. We have confirmed that this issue is isolated to requests using invalid or expired credentials and that backend services and Connected Search functionality remain fully operational for all other clients. We are actively reproducing the exact credential behavior to resolve the disruption. Further updates will be provided as additional details become available.

3. resolved Aug 06, 2026, 04:19 PM UTC

   Connected Search and the Lucidworks Platform are fully operational, having experienced no service degradation. The root cause was confirmed to be isolated to a single customer issuing search requests using invalid credentials. The issue was validated and resolved after confirming all backend systems and platform functionality remained operating normally. Other customers utilizing the Lucidworks Platform are experiencing no impact. Please contact Lucidworks Support (https://support.lucidworks.com/hc/en-us) for any questions or assistance needed.

Read the full incident report →

Started Jul 27, 2026, 06:19 PM UTC· resolved Jul 27, 2026, 07:24 PM UTC

1. investigating Jul 27, 2026, 06:19 PM UTC

   The Lucidworks SaaS Platform is currently experiencing a widespread service disruption affecting administrative controls. Users are unable to access the SaaS Platform UI and are receiving certificate error responses. End users are not impacted, as all other services remain operational. We are aware of the underlying issue and are actively working to rectify it. We will provide further updates as soon as that fix is in place.

2. identified Jul 27, 2026, 06:22 PM UTC

   We are aware of the underlying issue and are actively working to rectify it. We will provide further updates as soon as that fix is in place.

3. identified Jul 27, 2026, 06:53 PM UTC

   As the Lucidworks Signals Beacon sends data to api.lucidworks.com, it is affected more significantly by this incident than other products. Analytics Studio customers may see a gap in signals data collection through the duration of this incident. We will provide another update here as soon as the certificates have been replaced.

4. monitoring Jul 27, 2026, 07:03 PM UTC

   New certificates have been configured to mitigate the disruption, and internal testing has confirmed that all services are stable and available. Lucidworks services and software are once again fully operational. End users are experiencing no impact. We are continuing to actively monitor system performance to ensure ongoing stability before providing a final resolution update.

5. resolved Jul 27, 2026, 07:24 PM UTC

   The incident has now been fully resolved and validated. All previously-affected Lucidworks services and software are operating normally. Platform users can access the SaaS Platform UI without encountering errors. Signals Beacon data collection is once again functioning as well. The service disruption was caused by an expired SSL certificate. To resolve the issue, new certificates were successfully deployed. A postmortem report will be shared here within 1 business day.

6. postmortem Jul 28, 2026, 07:54 PM UTC

   # Summary On July 27, 2026, at 17:56 UTC, the Secure Sockets Layer/Transport Layer Security \(SSL/TLS\) certificate serving the Lucidworks SaaS Platform domains expired. Users attempting to access the SaaS Platform UI at [platform.lucidworks.com](http://platform.lucidworks.com) received certificate errors, and Beacon signal collection via [api.lucidworks.com](http://api.lucidworks.com) was interrupted. Query serving by underlying search services was not affected, and Lucidworks AI continued to function. Investigation confirmed that the certificate covering [platform.lucidworks.com](http://platform.lucidworks.com), [api.lucidworks.com](http://api.lucidworks.com), and an internal domain had reached its expiration date without a replacement in place. The certificate was managed through a manual renewal process. Advance expiration warnings did not reach the responsible team, in part due to a delivery failure in the alert-notification pipeline, which is currently under investigation with the relevant vendors. Lucidworks generated a replacement certificate, completed domain ownership validation, and deployed the new certificate to the affected load balancers. Lucidworks engineers confirmed the new certificate was serving and verified that all affected services had returned to normal operation. Following a period of stable monitoring, the incident was confirmed to be fully resolved at 19:24 UTC. # Root Cause TLS connections to [platform.lucidworks.com](http://platform.lucidworks.com), [api.lucidworks.com](http://api.lucidworks.com), and an internal Production domain are secured by a shared Subject Alternative Name \(SAN\) certificate. This certificate was historically issued through a third-party certificate authority and renewed through a manual rotation process. Other Lucidworks domains and services use separate certificates and were not affected. The certificate reached its expiration date on July 27, 2026, at 17:56 UTC without a replacement having been issued. When the certificate expired, TLS connections to the affected domains failed, rendering the SaaS Platform UI inaccessible and interrupting Beacon signal ingestion. Advance warning of the expiration existed but did not result in action. A synthetic monitoring test generated a 30-day advance alert on June 27, 2026, which was delivered to an internal alert channel but was not acted upon. The monitoring test continued to run and fire as designed, but beginning on July 1, 2026, alerts stopped being delivered from the monitoring platform to the downstream incident management system, so the scheduled follow-up notifications and pending outage escalations never reached the responsible team. This delivery failure was silent — no monitoring existed on the alert pipeline itself — and is under active investigation with the vendors of our monitoring platform and incident management system. Review of the response itself found no platform defect. When the expired certificate was identified, replacement issuance, validation, and deployment proceeded without error. As part of the replacement, Lucidworks changed certificate providers rather than renewing through the incumbent certificate authority. Because this was the first certificate issued through the new provider for these domains, Domain Name System \(DNS\) domain ownership validation was required before the certificate could be issued. This one-time validation requirement increased Lucidworks’ overall time to resolve the issue. Subsequent issuance through this provider will not require repeating this validation step. # Lucidworks Actions Lucidworks has taken or will take the following actions as a result of this incident: * Eliminate the manual rotation step entirely * Determine the root cause and fix the alert-delivery failure between the monitoring platform and the incident-management system, in coordination with both vendors * Escalate certificate expiration alerts to a paged alerting path that requires acknowledgment and is separated from lower-urgency notification traffic # Recommended Client Actions No client action is required. Signals sent to the Beacon endpoint during the incident window \(17:56–19:01 UTC on July 27, 2026\) were not ingested and are not recoverable. Analytics for this period may reflect a corresponding gap in data or dip in graphs. Clients with questions or observations not described in this report are encouraged to contact Lucidworks Support.

Read the full incident report →

Started Jun 19, 2026, 04:23 PM UTC· resolved Jun 19, 2026, 05:56 PM UTC

1. investigating Jun 19, 2026, 05:08 PM UTC

   Lucidworks Commerce Studio is currently experiencing intermittent latency, specifically affecting the Rules component. Customers attempting to interact with the Rules interface will experience timeouts when trying to load their data. There is no direct end-user impact, as this issue only affects internal administrative tasks within the Lucidworks Platform. We are actively investigating the cause of this disruption and will provide further updates as more information becomes available.

2. identified Jun 19, 2026, 05:21 PM UTC

   Commerce Studio is experiencing reduced performance within the Rules API due to apparent underlying database contention. Platform users attempting to interact with the Rules section of the interface, particularly when selecting the option to display 100 rules per page, will encounter timeouts. There is no direct end-user impact to search experiences, as this issue is confined to administrative functions within the management console. We believe the issue to be caused by database contention specifically triggered by simultaneous interactions involving a large number of rules. We are currently scaling up the database to alleviate the contention. While this scaling process is underway, some page loads may be interrupted. We are continuing to work toward full resolution and will provide an update once the system has stabilized.

3. monitoring Jun 19, 2026, 05:48 PM UTC

   We have scaled up the database to alleviate the performance issues. Performance has now normalized, and Commerce Studio Rules interactions have stabilized. Platform users may still observe minor latency when attempting to page through 100 rules at a time within the interface, but these requests no longer experience complete timeouts or produce errors. There is no direct end-user impact. We are continuing to monitor database response times and page load times, and we will provide a final update once full resolution is confirmed.

4. resolved Jun 19, 2026, 05:56 PM UTC

   The issue affecting the Commerce Studio Rules interface has been resolved, and all previously affected services are now operating normally. Commerce Studio users can interact with the Rules interface and page through large datasets without experiencing page load errors. There remains no direct end-user impact on search experiences. The root cause of the incident was database contention triggered by interactions involving a large number of rules. To resolve the issue, we scaled up the database, which normalized performance and stabilized the system.

Read the full incident report →

Started May 28, 2026, 05:14 PM UTC· resolved May 28, 2026, 06:08 PM UTC

1. investigating May 28, 2026, 05:14 PM UTC

   Certain Lucidworks AI hosted models (llama-3-8b-instruct, llama-3v2-3b-instruct, and phi-4-multimodal-instruct) in the us-southcarolina region are experiencing a service disruption, causing them to be currently unavailable. Customers attempting to use these Lucidworks-hosted models are receiving 500 or 429 errors for queries that rely on them. End users utilizing services that depend on these specific models are encountering errors, though passthrough LLM calls remain fully functional. We are currently investigating the issue to determine the cause and are actively working to restore full availability. We will provide further updates as new information becomes available.

2. identified May 28, 2026, 05:39 PM UTC

   The service disruption affecting certain Lucidworks AI hosted models remains ongoing, and the models are currently unavailable. The disruption occurred during routine Kubernetes maintenance involving node upgrades, during which a cloud provider capacity shortfall prevented the instances hosting these models from re-launching. We have secured the needed capacity and have engaged our cloud provider to resolve the remaining delays in bringing the models back up. We will provide further updates as additional information becomes available.

3. monitoring May 28, 2026, 06:01 PM UTC

   We have successfully launched new nodes with the required LLM images, and redirected Lucidworks AI traffic to them. All LWAI hosted models are fully functional again, and stability has been confirmed through successful 200 responses from the prediction endpoint. End-user functionality is fully restored, with queries no longer returning errors. We are continuing to observe monitoring metrics to ensure system stability remains consistent before providing a final resolution update.

4. resolved May 28, 2026, 06:08 PM UTC

   The service disruption affecting certain Lucidworks AI hosted models has been fully resolved, and all previously affected services are operating normally. End-user functionality is fully restored, with all dependent services operating as expected. The disruption originated during routine Kubernetes maintenance involving node upgrades, during which a cloud provider capacity shortfall prevented the instances hosting these models from re-launching. Stability was restored after successfully launching new nodes and pulling the required images. We will share a postmortem report containing the full root cause analysis within three business days.

5. postmortem Jun 02, 2026, 10:33 PM UTC

   ## Summary On May 28, 2026, between 16:18 UTC and 17:53 UTC, Lucidworks AI hosted LLMs were unavailable in the `us-southcarolina` region. Customers using hosted LLM inference \(`llama-3-8b-instruct`, `llama-3v2-3b-instruct`, and `phi-4-multimodal-instruct`\) received 500 or 429 errors when attempting to query these models. Other SaaS Platform services, including search, embedding models, and the Lucidworks Platform UI, were not affected. Lucidworks Engineering declared a Sev1 incident at 16:54 UTC and began remediation efforts. All hosted LLM models were fully restored and operational at 17:53 UTC. ## Root Cause The incident was caused by a routine Kubernetes patch upgrade on a cluster in the `us-southcarolina` region. LWAI-hosted models are served via [Ray Serve](https://docs.ray.io/en/latest/serve/index.html), which uses both “head” and “worker” nodes as part of its deployment system for routing inference requests. The Kubernetes upgrade cycled node pools, causing all cluster head nodes and worker nodes to restart simultaneously. Under normal conditions, the Ray cluster can tolerate a head node restart because worker nodes continue serving requests. However, the node pool upgrades utilize a surge strategy where our platform waits for pods to leave the old node \(be evicted\) but not for them to be running on the new node. The platform considers the “drain” successful once the pod is gone from the old node and moves on to the next one, even if the pod is stuck in an “Initializing” state on the new node. This meant that all node pools were cycled in rapid succession, and both head and worker pods were evicted before any had finished initializing on their replacement nodes, resulting in a complete cluster outage. Recovery was prolonged by multiple compounding factors. First, one of the replacement head nodes was in a degraded state and unable to pull container images, requiring manual intervention to delete the node. Second, the LLM container images \(6-11 GB in size\) experienced abnormally slow Docker image transfer, taking 30-52 minutes compared to the typical 2-4 minutes observed in normal operation. Additionally, in a separate operation, new models were being brought online to expand our LWAI offering, and this caused the Ray operator's blue-green deployment strategy to require the existing _and_ replacement LLM deployments to be healthy before switching traffic, which extended the outage until the slower image pulls completed on both blue and green deployments. Lucidworks Engineering deleted the degraded node, waited for image pulls to complete on replacement nodes, and verified that all hosted models were responding to queries. The incident was verified as resolved at 18:07 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Implement sequenced Kubernetes upgrade procedures for clusters hosting LLM workloads, ensuring each node pool is fully healthy before the next pool is upgraded. * Investigate Docker image pre-loading strategies \(such as pre-baked disks or image streaming\) to eliminate long container image pull times for large ML model images. * Open a support ticket with our cloud provider to investigate the abnormal Docker pull times. * Establish a notification protocol to coordinate Kubernetes maintenance windows with LLM service owners to avoid conflicts with ongoing deployments. * Improve tooling around our Ray clusters to allow Lucidworks Engineering to force a failover to a blue or green state instead of waiting for Ray to automatically resolve the new and old deployments. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).

Read the full incident report →

Started May 12, 2026, 09:19 PM UTC· resolved May 13, 2026, 12:01 AM UTC

1. investigating May 12, 2026, 11:04 PM UTC

   We are currently investigating an issue affecting the Lucidworks Platform user interface. While services hosted on the Platform remain operational and the Beacon continues to receive signals from client sites, all SaaS users are unable to access the SaaS Platform interface. Users attempting to load platform.lucidworks.com encounter an error stating that applications failed to load and menu items may be unavailable. There is no direct impact to the client’s end-user experience at this time. We are currently working to identify the cause of this disruption and will provide further updates as more information becomes available.

2. identified May 12, 2026, 11:23 PM UTC

   Lucidworks has identified that a recent configuration update resulted in the removal of a critical data component within our internal database. This discrepancy is preventing the SaaS Platform interface from retrieving the information required to load correctly, leading to the access errors currently being experienced by users. While services hosted on the Platform remain operational and Lucidworks AI and Beacon services continue to function normally, all SaaS users remain unable to access the SaaS Platform interface. We are currently implementing a database update to restore the missing components and are repopulating the necessary data from a secure backup to ensure all settings are accurately recovered. Users should refrain from attempting to reconfigure platform settings or applications until access is fully restored. We will provide a further update as soon as more information is available.

3. monitoring May 12, 2026, 11:55 PM UTC

   Engineering has successfully deployed a fix to restore the missing data and has been merged into the production environment. We are now observing a period of stability following these mitigation actions. The SaaS Platform interface is now accessible, and the previously reported errors have been cleared. Lucidworks AI and Beacon services remain stable and were unaffected throughout the event. No widespread user impact is currently being observed, though we are performing final data validation to ensure all configuration values have been fully restored across the environment. We are continuing to monitor platform stability and system logs to confirm that all services maintain normal performance levels. We will provide a final update once the monitoring period is complete and the incident is fully resolved.

4. resolved May 13, 2026, 12:01 AM UTC

   Lucidworks has confirmed the full resolution of this incident. The SaaS Platform interface is once again accessible, and the errors previously encountered at platform.lucidworks.com have been cleared. The Lucidworks Platform is operating normally for all users. Lucidworks AI and Beacon services remained stable and were not affected throughout this event. There is no lingering impact to platform users or end-user experiences. A postmortem report will be shared once the internal review is complete.

5. postmortem May 20, 2026, 09:05 PM UTC

   ## Summary On May 12, 2026, at 21:19 UTC, the Lucidworks SaaS Platform experienced an access disruption. Users encountered errors when attempting to use the Lucidworks Platform UI \([platform.lucidworks.com](http://platform.lucidworks.com)\). Other SaaS Platform services, including all search and Lucidworks AI functionality, were not affected. Lucidworks Engineering identified the issue at 21:25 UTC and began remediation efforts. The platform was fully restored and operational at 23:41 UTC. ## Root Cause The incident was caused by a database schema change that was included alongside an application code update. The Lucidworks Platform runs multiple instances of each application simultaneously, and new code is rolled out incrementally. New application instances are brought online while existing instances continue serving traffic. In this case, the database schema was updated before all application instances had received the new code, causing existing instances to query a column that no longer existed.The schema change removed a database column before all running instances of the application had been updated to use the new schema. During the rollout window, existing application instances attempted to query the removed column, resulting in errors that prevented users from accessing the platform. Lucidworks Engineering deployed a fix that restored the affected database column and repopulated the necessary data from the most recently-taken database backup. The incident was verified as resolved at 23:41 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Enforce phased rollout procedures for database schema changes, ensuring destructive modifications are deployed separately from application code updates to maintain backward compatibility through the rollout window * Update automated code review tooling to detect and flag destructive database operations, adding an additional layer of defense during the review process * Evaluate automated incident creation from critical alerts to reduce the time between detection and formal incident response The Lucidworks engineering team is committed to ensuring this type of incident does not recur. These enhancements will strengthen the platform's resilience and reliability for all customers. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).

Read the full incident report →

Started Apr 24, 2026, 04:05 PM UTC· resolved Apr 24, 2026, 04:13 PM UTC

1. resolved Apr 24, 2026, 04:34 PM UTC

   The Lucidworks SaaS Platform experienced an issue that caused a total disruption of service routing, rendering the platform inaccessible. Users were unable to log in or manage any platform configurations during this period. End users were unable to access search results or interact with any deployed services. The incident was caused by an unintended deployment of a configuration change at 4:06 UTC that should not have gone to production. We resolved the issue by performing a full rollback of the faulty change, which was completed at 4:13 UTC. A postmortem report will be posted with the full incident details.

2. postmortem May 18, 2026, 10:56 PM UTC

   ## Summary On April 24, 2026, at 16:05 UTC, the Lucidworks SaaS Platform experienced an issue that made the entire platform temporarily unavailable and all requests returned IO errors. Lucidworks Engineering was made aware of the issue and reverted the change on April 24, 2026, at 16:08 UTC. The reversion took a few minutes to propagate through the system and successful responses were restored by April 24, 2026, at 16:13 UTC. ## Root Cause The incident was caused by a configuration change to the SaaS Platform gateway components that was deployed to production at 16:05 UTC on April 24, 2026. This change introduced an error in the configuration template that caused it to render syntactically invalid output in the production environment. When the deployment system processed this invalid configuration, it interpreted the malformed output as an indication that the gateway components \(routing and load-balancing infrastructure\) were no longer needed and removed these components from the platform. This caused all incoming requests to fail with IO errors as there were no gateway instances available to handle traffic. Lucidworks engineers were actively monitoring this deployment as it rolled out, and immediately noticed the problem that the deployment introduced. The configuration change was quickly reverted at 16:08 UTC. The deployment infrastructure then correctly reinstated the gateway components, which became fully operational by 16:13 UTC and restored normal platform operation. ## Lucidworks Actions Lucidworks has taken the following actions as a result of this incident: * Added pre-deployment validation to verify that configuration templates render syntactically valid output before deployment, and that changes of this nature are deployed to a development environment before deploying the changes to production * Modified the deployment workflow to require additional verification steps before pull requests can be merged ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).

Read the full incident report →

Started Mar 06, 2026, 05:05 AM UTC· resolved Mar 06, 2026, 06:13 AM UTC

1. investigating Mar 06, 2026, 06:08 AM UTC

   We are currently experiencing a disruption of the SaaS Platform user interface. Users are receiving 404 errors when attempting to access platform.lucidworks.com. All other SaaS Platform services remain operational. There is no direct impact on end-user services or search experiences at this time, but the ability to provision or configure SaaS Platform-based services is affected. We are actively investigating the issue and taking steps to restore access. We will provide further updates as more information becomes available.

2. identified Mar 06, 2026, 06:09 AM UTC

   We believe this issue to have been caused by a recently-deployed change for a specific customer UI feature that is currently in development.

3. monitoring Mar 06, 2026, 06:10 AM UTC

   The recent change has been rolled back and UI functionality has been restored. We are monitoring to confirm full recovery.

4. resolved Mar 06, 2026, 06:13 AM UTC

   We've confirmed that all platform.lucidworks.com functionality has been restored after reverting the most recently-deployed change. This incident only involved customer access to the SaaS Platform user interface; no search or inference capabilities were affected, and all Platform services remained operational. A detailed analysis will be posted here as soon as it is available.

5. postmortem Mar 16, 2026, 05:51 PM UTC

   ## Summary On March 6, 2026, at 05:05 UTC, the Lucidworks SaaS Platform experienced a user interface disruption. Users were consistently redirected to a 404 error page when attempting to access [platform.lucidworks.com](http://platform.lucidworks.com). Other SaaS Platform services, including all search and LWAI functionality, were not affected. Lucidworks Engineering became aware of the issue at 05:40 UTC and the inaccessibility of the user interface was addressed by 05:45 UTC by reverting the most recently-deployed UI change. At 06:13 UTC, Lucidworks Engineering confirmed that access to the user interface had been fully restored and the platform was operational in its entirety. ## Root Cause The incident was caused by a configuration change to one of our currently in-development UI services that inadvertently affected routing across the entire platform. This change was carried out at 04:57 UTC. Following the code rollout, Lucidworks Engineering became aware that the SaaS Platform user interface was unavailable at 05:40 UTC during routine verification of this deployment. The deployment configuration setting for that UI service was left blank, which caused the service to claim routing priority for all platform traffic instead of only its intended path. The resulting global path misconfiguration caused the platform to become unavailable and redirected users to a 404 error page. Reverting this change restored the SaaS Platform to its previous configuration. The incident was verified as resolved at 06:13 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Establish clear guidelines for infrastructure-sensitive changes * Include more explicit warnings to configuration files for critical settings and update internal documentation to highlight high-risk configuration areas Additionally, Lucidworks will take the following actions to further enhance our ability to detect, withstand, and respond to similar incidents in the future: * Enhance automated pre-deployment testing by integrating validation tools that run before any deployment to ensure errors are visible to engineering teams earlier in the process * Enhance real-time monitoring for routing anomalies, adding alerts for unexpected traffic patterns, and creating dashboards to provide visibility into platform routing health The Lucidworks engineering team is committed to ensuring this type of incident does not recur. Lucidworks is implementing multiple layers of prevention, detection, and response improvements. These enhancements will strengthen our platform's resilience and reliability for all customers. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** on [status.lucidworks.com](http://status.lucidworks.com).

Read the full incident report →

Started Dec 03, 2025, 06:26 PM UTC· resolved Dec 03, 2025, 10:05 PM UTC

1. investigating Dec 03, 2025, 06:26 PM UTC

   We are currently investigating this issue.

2. identified Dec 03, 2025, 06:30 PM UTC

   The Lucidworks SaaS Platform is currently experiencing an unexpected outage. This impacts all services, including Lucidworks AI, Commerce Studio, Analytics Studio, and Connected Search. We are actively investigating the root cause and deploying resources to restore full functionality as quickly as possible. Next Update: We will post a full status update within 30 minutes. We apologize for the disruption and appreciate your patience.

3. identified Dec 03, 2025, 07:28 PM UTC

   The Lucidworks Platform remains unavailable due to an outage with a critical third-party service provider. Our teams are working directly with the vendor's engineers to restore connectivity and service stability as quickly as possible.

4. identified Dec 03, 2025, 08:16 PM UTC

   The Lucidworks SaaS Platform remains unavailable as our teams are still actively engaged with the third-party service vendor. We have successfully identified the root cause as a network infrastructure issue on the vendor's side, and we are closely collaborating with their engineering team as they prioritize and execute the necessary corrective actions to restore service stability and platform connectivity.

5. identified Dec 03, 2025, 08:48 PM UTC

   The Lucidworks SaaS Platform remains unavailable due to the external third-party issue. The vendor has successfully identified the root cause within their system and their engineering teams are now actively working on the corrective fix. We are closely monitoring their progress and maintaining direct communication. We will post our next status update within 30 minutes

6. monitoring Dec 03, 2025, 09:01 PM UTC

   The core issue has been successfully mitigated by our third-party vendor. The Lucidworks SaaS Platform is now entering a critical monitoring phase as our teams verify full stability and functionality across all services (Lucidworks AI, Commerce Studio, etc.). We anticipate a full resolution announcement shortly.

7. resolved Dec 03, 2025, 10:05 PM UTC

   The Lucidworks SaaS Platform is now fully restored and operational. The third-party network issue has been successfully fixed and full stability has been confirmed during our monitoring period. We thank you for your patience. A detailed postmortem will be posted here as soon as it is available.

8. postmortem Dec 12, 2025, 09:09 PM UTC

   ## Summary On December 3, 2025, at 13:30 UTC, the Lucidworks SaaS Platform experienced widespread service disruptions affecting multiple clients. This issue affected Lucidworks AI functionality \(including Neural Hybrid Search\), Commerce Studio and Analytics Studio accessibility, and Connected Search. API requests received HTTP `401` and `403` errors with `invalid or expired token` messages, and the [platform.lucidworks.com](http://platform.lucidworks.com) UI became inaccessible. During this time, Lucidworks infrastructure also experienced a high volume of malicious traffic from external sources. Lucidworks Engineering resolved the issue by 22:03 UTC on December 3, 2025, restoring service for all affected products. A similar issue occurred on December 5, 2025, beginning at 16:17 UTC. Following further coordination with our third-party identity provider \(explained in detail below\), this repeat incident was resolved by 19:00 UTC that day. ## Root Cause The incident was caused by a combination of three interconnected issues that occurred simultaneously on December 3, 2025. Okta, Lucidworks' identity provider for Platform authentication, [experienced a service outage](https://status.okta.com/#incident/a9CWR00000012EI2AY), and in response implemented aggressive rate-limiting measures. During this period, Lucidworks Platform IP addresses were blocked by Okta for an extended duration, preventing the Lucidworks authentication system from retrieving JSON Web Key Sets \(JWKS\) from Okta's servers. This resulted in HTTP `401` `invalid/expired` token errors for authentication requests. Concurrent with the Okta outage, Lucidworks Platform infrastructure experienced a distributed denial of service \(DDoS\) attack in the form of a high rate of malicious requests from multiple geographic sources. The attack traffic significantly exceeded Okta's rate limit thresholds and contributed to Okta's decision to block the Lucidworks Platform IP addresses. Additionally, for some Lucidworks Search customers, [a configuration flag \(failOnError\)](https://doc.lucidworks.com/docs/5.9/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query#fail-on-error) in the Core Package vectorization stage was inadvertently left enabled in the Production environment following earlier debugging activities. When the Okta authentication issue prevented access to Lucidworks AI services, queries using Neural Hybrid Search failed completely rather than falling back to lexical-only search, amplifying the impact for clients using Neural Hybrid Search functionality. Lucidworks determined the root causes through analysis of Okta service status, review of authentication logs showing JWKS retrieval failures, traffic analysis identifying the DDoS attack pattern, coordination with Okta support, and code review of affected query pipeline configurations. Changing the Lucidworks Platform IP address immediately restored authentication functionality, confirming that Okta's IP blocking was the primary cause of the authentication failures. Two days later, on December 5, 2025, the Lucidworks Platform again began to experience `401` invalid/expired token errors for authentication requests. We again had to coordinate with Okta support to update their IP address allowlist in order to ensure our traffic was not blocked or unnecessarily rate-limited. ## Lucidworks Actions Lucidworks has taken the following actions as a result of this incident: * Implemented enhanced rate limiting and DDoS protection and firewall rules to defend against future attacks * Modified affected client query pipelines to disable the `failOnError` flag and enable automatic lexical-only fallback when vectorization services are unavailable * Changed the Lucidworks Platform IP address to restore authentication services and coordinated with Okta to maintain proper IP allowlisting * Escalated with Okta support to resolve the IP blocking issue and ensure continuity of authentication services Additionally, we intend to take the following actions to further enhance our ability to detect, withstand, and respond to similar incidents in the future: * Ensure debug configuration flags are removed from all Production query pipelines to prevent similar failures during future service disruptions * Implement caching mechanisms for JWKS to provide fallback authentication capabilities during Okta service disruptions * Enhance our firewall’s adaptive protection capabilities to more quickly detect and mitigate DDoS attacks * Implement improved synthetic monitoring with automated incident detection to reduce time to detection for similar issues * Establish a process to proactively notify Okta when Lucidworks Platform IP addresses change to maintain current allowlist configurations ## Recommended Client Actions Lucidworks recommends that all clients using Neural Hybrid Search upgrade to Fusion 5.9.15 or later as soon as possible. This version includes enhanced failsafe fallback mechanisms in the Neural Hybrid Query stage that automatically switch to lexical-only queries when Lucidworks AI services are unreachable, improving overall system resilience during service disruptions. Lucidworks also recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** on [status.lucidworks.com](http://status.lucidworks.com).

Read the full incident report →

Started Oct 30, 2025, 03:40 PM UTC· resolved Oct 30, 2025, 05:12 PM UTC

1. investigating Oct 30, 2025, 04:55 PM UTC

   We are experiencing an elevated rate of search errors for Connected Search customers. Many queries are currently returning 504 responses and the calls are unable to complete. Lucidworks personnel are actively investigating.

2. identified Oct 30, 2025, 04:57 PM UTC

   We have identified a recent change that seems to be the source of this issue, and are preparing to roll that change back now.

3. monitoring Oct 30, 2025, 05:01 PM UTC

   The change has been rolled back and we are starting to see recovery. Search queries for Connected Search endpoints are once again responding successfully. Our team is continuing to monitor to ensure all errors resolve.

4. resolved Oct 30, 2025, 05:12 PM UTC

   Rolling back the recent change has resolved this incident, and we have confirmed that we are no longer seeing query errors. Individual Connected Search customers who we know to have been affected have already had a proactive ticket raised by Lucidworks Support. Any additional inquiries or follow-ups can be submitted by visiting our Support Portal: https://support.lucidworks.com/hc/en-us We will post a full postmortem here within 48 hours.

5. postmortem Nov 01, 2025, 07:17 PM UTC

   # Summary On October 29, 2025, beginning at 15:40 UTC, Lucidworks Connected Search query responses began to produce an increased amount of 504 gateway timeout errors which affected the ability to serve search traffic. Lucidworks mitigated the issue by rolling back the most recent code change, resolving the incident at 17:01 UTC. Full functionality was verified to be restored by 17:12 UTC. The total impact duration was approximately 81 minutes, during which time users experienced intermittent increased search latency and failures in Connected Search environments. Lucidworks proactively opened outbound Support cases for affected customers during this incident. # Root Cause The incident was initially believed to be caused by a recent change to an unrelated service that introduced unstable connection handling. Following that update, we detected frequent connection resets to an upstream third-party service, which propagated to other services, resulting in timeouts and degraded search performance. A Severity 1 \(S1\) event was declared, and we began posting updates to our status page. Despite our belief that the most recently-deployed change was isolated from Connected Search, we reverted it in an attempt to restore service. Around the time that rollback completed, we began to see recovery and declared the incident resolved. However, in the process of debugging the change that was rolled back as part of our standard postmortem process, our testing suggests that the code in question may not have actually been the underlying source of this incident, and we are coordinating with one of our third-party providers to obtain more information. This incident report will be updated when that information is received. ### Update from 6 November 2025: After extensive investigation, consultation with third-party providers, and targeted testing of the suspected faulty code, Lucidworks identified the root cause as a compound issue. The recent change previously mentioned, designed to improve vector-lookup performance, successfully achieved that goal but inadvertently exposed a latent defect in HTTP connection reuse. In addition, the vector service was under-resourced, causing rapid scaling of replicas under query load. This scaling behavior increased the number of concurrent connections. This behavior, combined with lacking connection reuse, led to all available ports being exhausted. As a result, outgoing packets were dropped, and new connections could not be established to reach the index in order to serve search traffic. To resolve the issue, Lucidworks increased service resource allocations, explicitly implemented HTTP connection reuse, and added enhanced logging and validation safeguards to detect similar issues in the future. # Lucidworks Actions Lucidworks has taken the following actions as a result of this incident: * Rolled back the most recent change to the relevant service in production. This change will not be re-deployed until we’ve confirmed beyond a doubt that it will not cause a recurrence of this issue. * Restarted affected services to clear residual connection issues. * Added an internal process review to ensure change isolation assumptions are verified prior to all deployments. * Reached out to our third-party provider to get additional information and to determine if an unreported service issue on their end was the true underlying cause of our Connected Search traffic issues. ### Additional Lucidworks Actions following further root-cause analysis: * Improved our vector-serving backend to scale more gradually. * Implemented HTTP connection reuse where it was lacking. * Added timeouts and more rigorous logging for better visibility. * Scale-tested these changes at 100x normal QPS levels to ensure the issue was fully resolved. # Recommended Client Actions There are no recommended client actions as a result of this incident.

Read the full incident report →

Started Sep 08, 2025, 11:54 AM UTC· resolved Sep 08, 2025, 01:15 PM UTC

1. investigating Sep 08, 2025, 12:57 PM UTC

   We are seeing an elevated error rate on search requests for our Connected Search customers. Lucidworks Engineering is actively investigating the issue, and updates will be posted here as soon as we have more information to share.

2. investigating Sep 08, 2025, 01:00 PM UTC

   We are continuing to investigate this issue. We are seeing elevated query latency related to a critical backend search service, and are focusing our investigation there.

3. monitoring Sep 08, 2025, 01:05 PM UTC

   The issue was due to a performance degradation with one of our 3rd-party service providers. Once their service was fully recovered, we saw recovery of the elevated Connected Search error rate. We will continue to monitor this issue to ensure it is fully resolved.

4. resolved Sep 08, 2025, 01:15 PM UTC

   Recovery of the 3rd-party service provider's system led to recovery of our own backend search service, and Connected Search query responses have been fully functional since that time.

Read the full incident report →

Started May 26, 2025, 06:16 PM UTC· resolved May 26, 2025, 08:28 PM UTC

1. investigating May 26, 2025, 07:09 PM UTC

   We have been alerted to Lucidworks SaaS Platform errors being returned in response to ingestion and query API calls for Connected Search clients, as well as for LWAI calls coming via Managed Fusion Integrations. The Platform UI continues to function, and configuration changes can still be made. The Lucidworks team is investigating.

2. identified May 26, 2025, 08:10 PM UTC

   The specific errors we’re seeing are “SSL handshake fails for connections to *.applications.lucidworks.com.” Our team is pursuing all avenues of investigation with a focus on TLS certificate configurations for endpoints in the applications.lucidworks.com DNS domain.

3. monitoring May 26, 2025, 08:23 PM UTC

   We have identified a misconfiguration related to the TLS certificate configured for the applications.lucidworks.com domain. This configuration has been reverted and the SSL handshake errors we had been seeing have stopped. We are monitoring to confirm stability.

4. resolved May 26, 2025, 08:28 PM UTC

   We’ve confirmed that all Lucidworks Platform functionality related to this incident has been fully restored. We will share more details in a postmortem update as soon as our analysis is complete.

5. postmortem May 29, 2025, 06:52 PM UTC

   ## Summary On May 26, 2025, at approximately 17:50 UTC, clients began experiencing errors when attempting to access various parts of the Lucidworks platform, including LWAI and Connected Search. The incident was not immediately detected by our internal monitoring and alerting system. It was identified only after user reports and internal investigation, delaying the time to resolution. A recent change to the TLS certificate configuration used by these services was reverted at 20:04 UTC, resolving the issue. ## Root Cause The Lucidworks Platform utilizes several domain name spaces \(zones\) to provide its services. Critically, the `*.applications.lucidworks.com` zone houses addressable endpoints for all Applications powered by the Platform, including LWAI Applications and their associated Integrations \(which are used by Managed Fusion for intra-Platform communication\). During a routine update intended to replace an expiring certificate for `*.datasources.lucidworks.com `– an unrelated zone – the engineering team inadvertently deployed this certificate to several additional subdomains, including `*.applications.lucidworks.com`. Because the newly deployed certificate was only valid for `*.datasources.lucidworks.com`, services such as `*.applications.lucidworks.com` began serving an invalid certificate for their domain. This misalignment led to backend connection failures and disrupted user access to key parts of the Lucidworks Platform, including LWAI Applications and their associated Integrations, as well as Connected Search functionality. Lucidworks Engineering corrected the issue by rolling back the incorrect certificate and redeploying the correct certificates to all affected subdomains. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * **Automate Certificate Renewal**: We will implement an automated certificate renewal and deployment pipeline to remove manual steps and reduce the risk of human error. * **Enhanced Synthetic Testing:** We will expand synthetic monitoring coverage to include all Lucidworks subdomains, with continuous validation of TLS certificate alignment and service accessibility. While such monitoring is already in place, this incident has revealed that it is not currently extensive enough to capture all potential failure scenarios, so we are actively working to enhance it to ensure it will do so in the future. ## Recommended Client Actions There are no recommended client actions as a result of this incident.

Read the full incident report →

Started Apr 10, 2025, 03:43 PM UTC· resolved Apr 10, 2025, 04:19 PM UTC

1. investigating Apr 10, 2025, 04:28 PM UTC

   The Lucidworks Platform UI at platform.lucidworks.com is currently unavailable. This has no effect on product functionality other than users' ability to change configurations. Our teams are actively working to resolve the issue.

2. resolved Apr 10, 2025, 04:30 PM UTC

   A change related to enabling a new product feature was rolled out that had the unintended effect of making our Platform UI unavailable, which was not caught in our development and testing environment. That change has been rolled back, and the UI is once again available. No other product functionality was impacted by this incident. We will share more details in a postmortem update as soon as our analysis is complete.

3. postmortem Apr 17, 2025, 10:15 PM UTC

   ## Summary On April 10, 2025, starting at approximately 15:43 UTC, users of the Lucidworks Platform UI at [platform.lucidworks.com](http://platform.lucidworks.com) may have experienced issues accessing or altering their Lucidworks Platform product configurations. This was due to an error introduced during the deployment of a new service as part of a forthcoming product rollout. The issue was identified and resolved by 16:19 UTC. ## Root Cause The incident occurred when a new service was deployed with an incorrect network configuration. The Lucidworks Platform uses a global load-balancing and routing system to direct traffic to the various services it provides. When introducing a new route to accommodate a newly-developed service, a misconfiguration of that routing layer inadvertently resulted in UI traffic being directed to this new service, which affected the reachability of [platform.lucidworks.com](http://platform.lucidworks.com) for all users. Our engineering team identified the problem and resolved it by reverting the routing configuration to the previous stable state. Our internal investigation revealed that the configuration error stemmed from incomplete internal documentation as well as a need for more rigorous processes for deploying new services. We are taking steps to address these internal factors to prevent similar issues in the future, as described below. Detection of this incident could have been more timely. Our postmortem analysis revealed a misconfigured alert that was incorrectly getting set to a lower severity than is appropriate for this level of impact. This misconfiguration has been corrected. ## Lucidworks Actions Lucidworks is committed to providing a stable and reliable platform experience for all users. As a result of this incident, we are taking the following actions: * We have updated our internal documentation and tooling to make it clearer to our engineers how to introduce a new service onto the Platform without affecting any others. * We are implementing stricter controls and validation processes for new service deployments, to ensure they are first rolled out in our dev environment for functional validation, and to confirm their introduction does not affect any existing services, prior to deploying them into production. * We are also enhancing the way we manage routing configurations on the Platform in order to make all routing rules more strict, to ensure it will not be possible for traffic intended for one service to be inadvertently routed to a different service in the future. * As mentioned above, we have fixed a misconfigured alert to appropriately notify our teams with the highest severity, should something similar to this incident happen again in the future. ## Recommended Client Actions There are no recommended client actions as a result of this incident. If you have any follow-up questions or concerns, please submit a request with [Lucidworks Support](https://support.lucidworks.com/).

Read the full incident report →