Userpilot Outage History

All services have now recovered and are operating normally. We will continue to monitor the platform closely to ensure stability. Thank you for your patience while we worked through this issue.

The issue has been resolved. Ingestion throughput has returned to normal operating levels, and all data is being processed as expected.

As a follow-up, we isolated the affected subsystem under its own supervisor so that any similar external change or failure will be contained without affecting regional service availability.

A fix has been implemented and deployed, and we are currently monitoring system performance to ensure stability. We will share the postmortem for this incident as a follow-up message. Thank you for your patience.

Postmortem The incident was caused by our database background optimization process not running aggressively enough. This allowed data fragments to accumulate across tables used for real-time cache loading. As the number of fragments increased, queries that normally completed in milliseconds were forced to scan across many more fragments than necessary. This significantly increased query latency and led to exhaustion of available database connections, which impacted data ingestion and content publishing performance. To resolve this, we reconfigured our database to optimize these tables more frequently and in smaller batches. This keeps fragment counts low and ensures real-time queries remain fast and stable. Additionally, we have added monitoring and alerting on fragment count and size per table so we can detect abnormal accumulation early and prevent similar incidents in the future.

The issue causing delays in data ingestion has been fully resolved. Ingestion throughput has returned to normal operating levels, and all data is being processed as expected. There was no data loss during the incident. We will publish a post-mortem with additional details and preventive actions in a follow-up communication. Thank you for your patience and understanding.

The issue causing delays in data ingestion has been fully resolved. Ingestion throughput has returned to normal operating levels, and all data is being processed as expected. There was no data loss during the incident. We will publish a post-mortem with additional details and preventive actions in a follow-up communication. Thank you for your patience and understanding.

We have switched CDN providers temporarily. Our services are no longer affected by Cloudflare’s outage and are functioning normally. Thank you for your patience and understanding.

Web Application recovered

SSO Service has recovered. Our team continues to monitor the situation with our cloud provider.

We’ve taken steps to address the service degradation and are closely monitoring performance. Post Mortem What happened: After upgrading our columnar database, an incompatibility stopped automatic consolidation on a core table. Impact: Partial data loss and content triggering issues. Cause: The new version didn’t work correctly with tables that use pre-computed summaries, leading to too many small data fragments. Fix: Rebuilt the summaries to force consolidation and rolled back to the prior database version. Performance recovered. Prevention: Added an alert on fragment count, paused similar upgrades, and expanded compatibility/performance testing before future releases. We’re sorry for the disruption and appreciate your patience.