Entitle experienced a notice incident on April 10, 2025 affecting Access Change Requests, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved Apr 10, 2025, 03:09 PM UTC
This incident has been resolved.
- postmortem Apr 10, 2025, 03:09 PM UTC
### **Postmortem: Elastic Reindex Failure Impacting Sourcegraph Requests** **Date of Incident:** April 9, 2025 **Reported by:** Sourcegraph Integration **Impact Duration:** Until reindex was manually retriggered **Status:** Resolved ### **Summary** On April 9, 2025, a user from Sourcegraph encountered an error while attempting to create a new request. This was due to a failure in our daily Elasticsearch reindex process, which is responsible for refreshing role-based access data across integrations and bundles. ### **Root Cause** The reindex process failed due to an unusually large volume of roles on that day. This overwhelmed Elasticsearch, leading to internal failures and restarts. As a result: * Elasticsearch temporarily blocked access to the affected index. * Sourcegraph, which relies on this index to retrieve asset information and validate access, received errors when making requests. ### **Resolution** The issue was resolved by manually re-running the reindex operation, which successfully completed and restored access. ### **Impact** * Sourcegraph users were unable to create new requests during the failure window. * System stability and access to critical index data were temporarily compromised. ### **Action Items** 1. **Improve Monitoring on Elasticsearch Reindexing** * Add alerts for failed or unusually long reindexing jobs. 2. **Optimize the Reindex Process** * Investigate batching or parallelization strategies to better handle high-volume role data. 3. **Enhance Elasticsearch Stability and Fault Tolerance** *