Userpilot experienced a minor incident on October 14, 2025 affecting Current status by service (Event Ingestion (US - Standard)), lasting 1h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 14, 2025, 03:27 PM UTC
We are experiencing service degradation. Our team is actively monitoring and working to restore full performance.
- resolved Oct 14, 2025, 05:08 PM UTC
We’ve taken steps to address the service degradation and are closely monitoring performance. Post Mortem What happened: After upgrading our columnar database, an incompatibility stopped automatic consolidation on a core table. Impact: Partial data loss and content triggering issues. Cause: The new version didn’t work correctly with tables that use pre-computed summaries, leading to too many small data fragments. Fix: Rebuilt the summaries to force consolidation and rolled back to the prior database version. Performance recovered. Prevention: Added an alert on fragment count, paused similar upgrades, and expanded compatibility/performance testing before future releases. We’re sorry for the disruption and appreciate your patience.
- investigating Oct 14, 2025, 05:08 PM UTC
We’ve taken steps to address the service degradation and are closely monitoring performance. ## Post Mortem **What happened**: After upgrading our columnar database, an incompatibility stopped automatic consolidation on a core table. **Impact**: Partial data loss and content triggering issues. **Cause**: The new version didn’t work correctly with tables that use pre-computed summaries, leading to too many small data fragments. **Fix**: Rebuilt the summaries to force consolidation and rolled back to the prior database version. Performance recovered. **Prevention**: Added an alert on fragment count, paused similar upgrades, and expanded compatibility/performance testing before future releases. We’re sorry for the disruption and appreciate your patience.