Is Lucidworks down?

The Lucidworks SaaS Platform is currently experiencing a widespread service disruption affecting administrative controls. Users are unable to access the SaaS Platform UI and are receiving certificate error responses. End users are not impacted, as all other services remain operational.  We are aware of the underlying issue and are actively working to rectify it. We will provide further updates as soon as that fix is in place.

We are aware of the underlying issue and are actively working to rectify it. We will provide further updates as soon as that fix is in place.

As the Lucidworks Signals Beacon sends data to api.lucidworks.com, it is affected more significantly by this incident than other products. Analytics Studio customers may see a gap in signals data collection through the duration of this incident. We will provide another update here as soon as the certificates have been replaced.

New certificates have been configured to mitigate the disruption, and internal testing has confirmed that all services are stable and available. Lucidworks services and software are once again fully operational. End users are experiencing no impact. We are continuing to actively monitor system performance to ensure ongoing stability before providing a final resolution update.

The incident has now been fully resolved and validated. All previously-affected Lucidworks services and software are operating normally. Platform users can access the SaaS Platform UI without encountering errors. Signals Beacon data collection is once again functioning as well. The service disruption was caused by an expired SSL certificate. To resolve the issue, new certificates were successfully deployed. A postmortem report will be shared here within 1 business day.

# Summary On July 27, 2026, at 17:56 UTC, the Secure Sockets Layer/Transport Layer Security \(SSL/TLS\) certificate serving the Lucidworks SaaS Platform domains expired. Users attempting to access the SaaS Platform UI at [platform.lucidworks.com](http://platform.lucidworks.com) received certificate errors, and Beacon signal collection via [api.lucidworks.com](http://api.lucidworks.com) was interrupted. Query serving by underlying search services was not affected, and Lucidworks AI continued to function. Investigation confirmed that the certificate covering [platform.lucidworks.com](http://platform.lucidworks.com), [api.lucidworks.com](http://api.lucidworks.com), and an internal domain had reached its expiration date without a replacement in place. The certificate was managed through a manual renewal process. Advance expiration warnings did not reach the responsible team, in part due to a delivery failure in the alert-notification pipeline, which is currently under investigation with the relevant vendors. Lucidworks generated a replacement certificate, completed domain ownership validation, and deployed the new certificate to the affected load balancers. Lucidworks engineers confirmed the new certificate was serving and verified that all affected services had returned to normal operation. Following a period of stable monitoring, the incident was confirmed to be fully resolved at 19:24 UTC. # Root Cause TLS connections to [platform.lucidworks.com](http://platform.lucidworks.com), [api.lucidworks.com](http://api.lucidworks.com), and an internal Production domain are secured by a shared Subject Alternative Name \(SAN\) certificate. This certificate was historically issued through a third-party certificate authority and renewed through a manual rotation process. Other Lucidworks domains and services use separate certificates and were not affected. The certificate reached its expiration date on July 27, 2026, at 17:56 UTC without a replacement having been issued. When the certificate expired, TLS connections to the affected domains failed, rendering the SaaS Platform UI inaccessible and interrupting Beacon signal ingestion. Advance warning of the expiration existed but did not result in action. A synthetic monitoring test generated a 30-day advance alert on June 27, 2026, which was delivered to an internal alert channel but was not acted upon. The monitoring test continued to run and fire as designed, but beginning on July 1, 2026, alerts stopped being delivered from the monitoring platform to the downstream incident management system, so the scheduled follow-up notifications and pending outage escalations never reached the responsible team. This delivery failure was silent — no monitoring existed on the alert pipeline itself — and is under active investigation with the vendors of our monitoring platform and incident management system. Review of the response itself found no platform defect. When the expired certificate was identified, replacement issuance, validation, and deployment proceeded without error. As part of the replacement, Lucidworks changed certificate providers rather than renewing through the incumbent certificate authority. Because this was the first certificate issued through the new provider for these domains, Domain Name System \(DNS\) domain ownership validation was required before the certificate could be issued. This one-time validation requirement increased Lucidworks’ overall time to resolve the issue. Subsequent issuance through this provider will not require repeating this validation step. # Lucidworks Actions Lucidworks has taken or will take the following actions as a result of this incident: * Eliminate the manual rotation step entirely * Determine the root cause and fix the alert-delivery failure between the monitoring platform and the incident-management system, in coordination with both vendors * Escalate certificate expiration alerts to a paged alerting path that requires acknowledgment and is separated from lower-urgency notification traffic # Recommended Client Actions No client action is required. Signals sent to the Beacon endpoint during the incident window \(17:56–19:01 UTC on July 27, 2026\) were not ingested and are not recoverable. Analytics for this period may reflect a corresponding gap in data or dip in graphs. Clients with questions or observations not described in this report are encouraged to contact Lucidworks Support.

Lucidworks Commerce Studio is currently experiencing intermittent latency, specifically affecting the Rules component. Customers attempting to interact with the Rules interface will experience timeouts when trying to load their data. There is no direct end-user impact, as this issue only affects internal administrative tasks within the Lucidworks Platform. We are actively investigating the cause of this disruption and will provide further updates as more information becomes available.

Commerce Studio is experiencing reduced performance within the Rules API due to apparent underlying database contention. Platform users attempting to interact with the Rules section of the interface, particularly when selecting the option to display 100 rules per page, will encounter timeouts. There is no direct end-user impact to search experiences, as this issue is confined to administrative functions within the management console. We believe the issue to be caused by database contention specifically triggered by simultaneous interactions involving a large number of rules. We are currently scaling up the database to alleviate the contention. While this scaling process is underway, some page loads may be interrupted. We are continuing to work toward full resolution and will provide an update once the system has stabilized.

We have scaled up the database to alleviate the performance issues. Performance has now normalized, and Commerce Studio Rules interactions have stabilized. Platform users may still observe minor latency when attempting to page through 100 rules at a time within the interface, but these requests no longer experience complete timeouts or produce errors. There is no direct end-user impact. We are continuing to monitor database response times and page load times, and we will provide a final update once full resolution is confirmed.

The issue affecting the Commerce Studio Rules interface has been resolved, and all previously affected services are now operating normally. Commerce Studio users can interact with the Rules interface and page through large datasets without experiencing page load errors. There remains no direct end-user impact on search experiences. The root cause of the incident was database contention triggered by interactions involving a large number of rules. To resolve the issue, we scaled up the database, which normalized performance and stabilized the system.

Certain Lucidworks AI hosted models (llama-3-8b-instruct, llama-3v2-3b-instruct, and phi-4-multimodal-instruct) in the us-southcarolina region are experiencing a service disruption, causing them to be currently unavailable. Customers attempting to use these Lucidworks-hosted models are receiving 500 or 429 errors for queries that rely on them. End users utilizing services that depend on these specific models are encountering errors, though passthrough LLM calls remain fully functional. We are currently investigating the issue to determine the cause and are actively working to restore full availability. We will provide further updates as new information becomes available.

The service disruption affecting certain Lucidworks AI hosted models remains ongoing, and the models are currently unavailable. The disruption occurred during routine Kubernetes maintenance involving node upgrades, during which a cloud provider capacity shortfall prevented the instances hosting these models from re-launching. We have secured the needed capacity and have engaged our cloud provider to resolve the remaining delays in bringing the models back up. We will provide further updates as additional information becomes available.

We have successfully launched new nodes with the required LLM images, and redirected Lucidworks AI traffic to them. All LWAI hosted models are fully functional again, and stability has been confirmed through successful 200 responses from the prediction endpoint. End-user functionality is fully restored, with queries no longer returning errors. We are continuing to observe monitoring metrics to ensure system stability remains consistent before providing a final resolution update.

The service disruption affecting certain Lucidworks AI hosted models has been fully resolved, and all previously affected services are operating normally. End-user functionality is fully restored, with all dependent services operating as expected. The disruption originated during routine Kubernetes maintenance involving node upgrades, during which a cloud provider capacity shortfall prevented the instances hosting these models from re-launching. Stability was restored after successfully launching new nodes and pulling the required images. We will share a postmortem report containing the full root cause analysis within three business days.

## Summary On May 28, 2026, between 16:18 UTC and 17:53 UTC, Lucidworks AI hosted LLMs were unavailable in the `us-southcarolina` region. Customers using hosted LLM inference \(`llama-3-8b-instruct`, `llama-3v2-3b-instruct`, and `phi-4-multimodal-instruct`\) received 500 or 429 errors when attempting to query these models. Other SaaS Platform services, including search, embedding models, and the Lucidworks Platform UI, were not affected. Lucidworks Engineering declared a Sev1 incident at 16:54 UTC and began remediation efforts. All hosted LLM models were fully restored and operational at 17:53 UTC. ## Root Cause The incident was caused by a routine Kubernetes patch upgrade on a cluster in the `us-southcarolina` region.  LWAI-hosted models are served via [Ray Serve](https://docs.ray.io/en/latest/serve/index.html), which uses both “head” and “worker” nodes as part of its deployment system for routing inference requests.  The Kubernetes upgrade cycled node pools, causing all cluster head nodes and worker nodes to restart simultaneously.  Under normal conditions, the Ray cluster can tolerate a head node restart because worker nodes continue serving requests.  However, the node pool upgrades utilize a surge strategy where our platform waits for pods to leave the old node \(be evicted\) but not for them to be running on the new node. The platform considers the “drain” successful once the pod is gone from the old node and moves on to the next one, even if the pod is stuck in an “Initializing” state on the new node. This meant that all node pools were cycled in rapid succession, and both head and worker pods were evicted before any had finished initializing on their replacement nodes, resulting in a complete cluster outage. Recovery was prolonged by multiple compounding factors. First, one of the replacement head nodes was in a degraded state and unable to pull container images, requiring manual intervention to delete the node. Second, the LLM container images \(6-11 GB in size\) experienced abnormally slow Docker image transfer, taking 30-52 minutes compared to the typical 2-4 minutes observed in normal operation. Additionally, in a separate operation, new models were being brought online to expand our LWAI offering, and this caused the Ray operator's blue-green deployment strategy to require the existing _and_ replacement LLM deployments to be healthy before switching traffic, which extended the outage until the slower image pulls completed on both blue and green deployments. Lucidworks Engineering deleted the degraded node, waited for image pulls to complete on replacement nodes, and verified that all hosted models were responding to queries. The incident was verified as resolved at 18:07 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Implement sequenced Kubernetes upgrade procedures for clusters hosting LLM workloads, ensuring each node pool is fully healthy before the next pool is upgraded. * Investigate Docker image pre-loading strategies \(such as pre-baked disks or image streaming\) to eliminate long container image pull times for large ML model images. * Open a support ticket with our cloud provider to investigate the abnormal Docker pull times. * Establish a notification protocol to coordinate Kubernetes maintenance windows with LLM service owners to avoid conflicts with ongoing deployments. * Improve tooling around our Ray clusters to allow Lucidworks Engineering to force a failover to a blue or green state instead of waiting for Ray to automatically resolve the new and old deployments. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).

We are currently investigating an issue affecting the Lucidworks Platform user interface. While services hosted on the Platform remain operational and the Beacon continues to receive signals from client sites, all SaaS users are unable to access the SaaS Platform interface. Users attempting to load platform.lucidworks.com encounter an error stating that applications failed to load and menu items may be unavailable. There is no direct impact to the client’s end-user experience at this time. We are currently working to identify the cause of this disruption and will provide further updates as more information becomes available.

Lucidworks has identified that a recent configuration update resulted in the removal of a critical data component within our internal database. This discrepancy is preventing the SaaS Platform interface from retrieving the information required to load correctly, leading to the access errors currently being experienced by users. While services hosted on the Platform remain operational and Lucidworks AI and Beacon services continue to function normally, all SaaS users remain unable to access the SaaS Platform interface. We are currently implementing a database update to restore the missing components and are repopulating the necessary data from a secure backup to ensure all settings are accurately recovered. Users should refrain from attempting to reconfigure platform settings or applications until access is fully restored. We will provide a further update as soon as more information is available.

Engineering has successfully deployed a fix to restore the missing data and has been merged into the production environment. We are now observing a period of stability following these mitigation actions. The SaaS Platform interface is now accessible, and the previously reported errors have been cleared. Lucidworks AI and Beacon services remain stable and were unaffected throughout the event. No widespread user impact is currently being observed, though we are performing final data validation to ensure all configuration values have been fully restored across the environment. We are continuing to monitor platform stability and system logs to confirm that all services maintain normal performance levels. We will provide a final update once the monitoring period is complete and the incident is fully resolved.

Lucidworks has confirmed the full resolution of this incident. The SaaS Platform interface is once again accessible, and the errors previously encountered at platform.lucidworks.com have been cleared. The Lucidworks Platform is operating normally for all users. Lucidworks AI and Beacon services remained stable and were not affected throughout this event. There is no lingering impact to platform users or end-user experiences. A postmortem report will be shared once the internal review is complete.

## Summary On May 12, 2026, at 21:19 UTC, the Lucidworks SaaS Platform experienced an access disruption. Users encountered errors when attempting to use the Lucidworks Platform UI \([platform.lucidworks.com](http://platform.lucidworks.com)\). Other SaaS Platform services, including all search and Lucidworks AI functionality, were not affected. Lucidworks Engineering identified the issue at 21:25 UTC and began remediation efforts. The platform was fully restored and operational at 23:41 UTC. ## Root Cause The incident was caused by a database schema change that was included alongside an application code update. The Lucidworks Platform runs multiple instances of each application simultaneously, and new code is rolled out incrementally. New application instances are brought online while existing instances continue serving traffic. In this case, the database schema was updated before all application instances had received the new code, causing existing instances to query a column that no longer existed.The schema change removed a database column before all running instances of the application had been updated to use the new schema. During the rollout window, existing application instances attempted to query the removed column, resulting in errors that prevented users from accessing the platform. Lucidworks Engineering deployed a fix that restored the affected database column and repopulated the necessary data from the most recently-taken database backup. The incident was verified as resolved at 23:41 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Enforce phased rollout procedures for database schema changes, ensuring destructive modifications are deployed separately from application code updates to maintain backward compatibility through the rollout window * Update automated code review tooling to detect and flag destructive database operations, adding an additional layer of defense during the review process * Evaluate automated incident creation from critical alerts to reduce the time between detection and formal incident response The Lucidworks engineering team is committed to ensuring this type of incident does not recur. These enhancements will strengthen the platform's resilience and reliability for all customers. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).