Degraded Performance — Feature Flags in PROD2

Update timeline

1. investigating Apr 24, 2026, 07:16 PM UTC

   We are currently investigating this issue.

2. monitoring Apr 24, 2026, 07:29 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 24, 2026, 08:01 PM UTC

   This incident has been resolved.

4. postmortem Apr 30, 2026, 07:16 PM UTC

   ### Summary On April 24, 2026, a large non-batched bulk DELETE operation on the prod-2 primary database triggered lock contention, causing Feature Flag API latency and hung queries across multiple customer SDKs. ### Impact 1. Slow SDK auth/init — SDKs took longer than expected to complete evaluations 2. Elevated latency across many FF APIs 3. Limited to Feature Flag module, prod-2 4. No Data loss ### Root Cause A background cleanup job executed a non-batched, single-transaction delete causing lock contention and API latency spikes **Mitigation** Immediately terminated the offending queries. ### Next Steps / Action Items To prevent such issues from happening again. we are working on 1. Enhanced alerting and observability on long running queries. 2. Permanently replace large single-transaction delete pattern with smaller batched deletes