Entitle experienced a critical incident on February 26, 2025 affecting Entitle Portal and API and 1 more component, lasting 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 26, 2025, 09:51 AM UTC
We are investigating reports of degraded performance in our app. Our team is actively investigating and seeking a resolution, and we'll continue to update here.
- resolved Feb 26, 2025, 10:01 AM UTC
This incident has been resolved.
- postmortem Mar 16, 2025, 10:15 AM UTC
> **Root Cause:** > The deployment resulted in downtime due to a database migration step that took significantly longer than originally anticipated. As the deployment pipeline was configured to execute migrations synchronously prior to launching the new version, the prolonged migration caused the older version to terminate before the new version was fully ready to serve traffic, resulting in temporary unavailability. > **Resolution:** > To avoid recurrence, future deployments will separate database migrations from application releases, using incremental migration strategies or pre-migration scripts to ensure database readiness. Additionally, we've enhanced our monitoring to better estimate migration durations and introduced procedures to roll back or pause deployment if a migration exceeds expected durations. > **Follow-up Actions:** > > Implemented separate pre-deployment migration jobs with clearly defined thresholds for maximum allowable duration. Improved monitoring and alerting on migration performance and durations. Updated deployment procedures to allow manual approval for potentially long-running migrations.