Entitle experienced a major incident on February 26, 2025 affecting Access Change Requests, lasting 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 26, 2025, 10:21 AM UTC
We are currently investigating this issue.
- resolved Feb 26, 2025, 10:38 AM UTC
This incident has been resolved.
- postmortem Mar 11, 2025, 11:08 AM UTC
**Postmortem Report: Elastic Index Update Execution Failure When New Release Uploaded** **Date of Incident:** 2/26/25 **Summary:** When the release was uploaded, the indexing job took longer than expected to complete, preventing customers from requesting access to integrations. This delay led to inconsistencies in the system, including missing data and degraded performance. **Impact:** * Users were unable to access integrations. **Resolution:** The index job eventually completed after the release was uploaded. However, this delay is not a sustainable approach, as it can result in missing data and potential system disruptions. **Action Items:** 1. **Implement a Pre-Release Phase:** * Introduce a pre-release stage in the deployment pipeline where essential background jobs, including indexing, are executed before the full release upload. 2. **Automate Pre-Release Job Execution:** * Ensure that indexing jobs run automatically before the final release is pushed. * Implement monitoring/logging to verify job completion. 3. **Enhance Deployment Validation Checks:** * Define a checklist to confirm that all required jobs have been executed before proceeding with the release. 4. **Improve Alerts & Monitoring:** * Set up automated alerts if the indexing job is skipped or fails to execute. **Lessons Learned:** * Lack of a structured pre-release phase led to an avoidable issue. * Critical background jobs should be enforced as part of the deployment workflow. * Automation can help prevent manual errors and ensure smooth releases.