ADOC - Multiple Features Experiencing Downtime

Update timeline

1. investigating Mar 27, 2025, 09:29 AM UTC

   We are currently investigating this issue.

2. investigating Mar 27, 2025, 09:31 AM UTC

   We are continuing to investigate this issue.

3. identified Mar 27, 2025, 09:32 AM UTC

   The fix is being implemented. ETA: 3:45 PM IST

4. monitoring Mar 27, 2025, 09:37 AM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring Mar 27, 2025, 09:47 AM UTC

   We are continuing to monitor for any further issues.

6. resolved Mar 27, 2025, 10:27 AM UTC

   This incident has been resolved.

7. postmortem Mar 28, 2025, 10:22 AM UTC

   # Root Cause Analysis ## Catalog Service Migration Failure During 4.2.0 Release **Date of Incident:** March 27, 2025 09:29 UTC **Resolved**: March 27, 2025 10:27 UTC **Duration**: 58 minutes **Severity**: P2 ## Affected Services/Functionality * Catalog Service * Data Reliability Features: * Policy Listing Page * Asset Discovery Page ‌ # Incident Summary During the 4.2.0 release deployment, a database migration failure in the Catalog Service caused partial system degradation. The issue stemmed from a schema validation mismatch between the database layer and application ORM, specically around tenant\_id eld length validation. ## Impact: * Intermittent failures in Data Reliability features * Inconsistent UI rendering * Mixed-version service communication issues ‌ # Root Cause Analysis ## Primary Cause: Schema Validation Mismatch **Database vs. ORM Inconsistency:** | **Layer** | **Field** | **Constraint** | | --- | --- | --- | | Database | tenant\_id | VARCHAR\(64\) | | Exposed ORM | tenant\_id | @Column\(length=34\) | \*The migration failed when processing a tenant\_id with 37 characters ## Technical Breakdown: 1. Database Schema: Allows 64-character tenant\_ids 2. ORM Entity: ‌ @Table\(name = "tenants"\) ‌ class Tenant : IntIdTable\(\) \{ ‌ val tenant\_id = varchar\("tenant\_id", 34\) // Constraint ‌ \} 1. Migration attempted to process existing 35-character ID 2. ORM validation rejected the value before DB interaction The migration failed when encountering a tenant\_id exceeding 34 characters. ## Failure Chain 1. Migration script aborted due to ORM validation 2. Catalog Service pods failed initialization 3. Kubernetes maintained previous version pods 4. Version mismatch caused API inconsistencies ‌ # Detailed Incident Timeline ## Investigation Phase | **Time \(UTC\)** | **Duration** | **Action** | | --- | --- | --- | | 09:29 | | Deployment failure detected | | 09:32 | 3 mins | Team mobilisation | | 09:37 | 5 mins | Root cause identified | ## Hotx Development | **Time \(UTC\)** | **Duration** | **Action** | | --- | --- | --- | | 09:40 | | Hotfix development started | | 09:45 | 5 mins | Core x implemented | | 09:50 | 5 mins | Validation logic added | | 09:52 | 2 mins | 09:52 2m Unit tests completed | ## Build & Deployment | **Time \(UTC\)** | **Duration** | **Action** | | --- | --- | --- | | 09:55 | | CI pipeline triggered | | 09:58 | 3 mins | Build artifacts ready | | 10:00 | 2 mins | Staging deployment | | 10:10 | 10 mins | Staging verication | | 10:15 | 5 mins | Production rollout | | 10:25 | 10 mins | Full deployment | | 10:27 | 2 mins | Systems normal | ‌ # Resolution ## Corrective Actions: 1. Deployed hotfix with: 1. Updated ORM constraints \(64 chars\) 2. Additional migration validation 2. Verified all tenant\_id values in production 3. Validated cross-service compatibility ‌ # Preventive Measures | **Action Item** | **Owner** | | --- | --- | | Schema validation pre-checks | Data Eng | ‌ # Key Learnings **Testing Gap**: Need real-production-data migration testing