Degraded Indexing Performance Observed in Semantic Search

Update timeline

1. investigating Mar 03, 2026, 01:07 PM UTC

   We are currently observing a degraded performance issue with file uploads to Semantic Search. However, this is limited to new file uploads, file deletes and viewing recently uploaded files.

2. identified Mar 03, 2026, 03:06 PM UTC

   The underlying issue has been identified and a fix is now being worked on. The backlog of the impacted files is now being processed and the situation is gradually improving. We will update once the components are fully operational.

3. monitoring Mar 05, 2026, 12:54 PM UTC

   Disk usage across the Elasticsearch cluster has decreased and cluster health is now green. A backfill is running to ensure all projects are up to date, and the system is being monitored to confirm stability.

4. monitoring Mar 09, 2026, 11:01 AM UTC

   With some improvements made, we are continuing to monitor the incident.

5. resolved Mar 16, 2026, 05:59 PM UTC

   Following the implementation of several key improvements, we have successfully stabilized the westeurope-1 environment and returned the Elasticsearch cluster to a healthy Green status. While disk usage has been significantly reduced to optimal levels, a backfill process is currently running to ensure all project data is fully synchronized and up to date. We have completed an extended monitoring period to confirm system stability, and all services are now fully operational.