Userpilot Outage History

A fix has been implemented and deployed, and we are currently monitoring system performance to ensure stability. We will share the postmortem for this incident as a follow-up message. Thank you for your patience.

Postmortem The incident was caused by our database background optimization process not running aggressively enough. This allowed data fragments to accumulate across tables used for real-time cache loading. As the number of fragments increased, queries that normally completed in milliseconds were forced to scan across many more fragments than necessary. This significantly increased query latency and led to exhaustion of available database connections, which impacted data ingestion and content publishing performance. To resolve this, we reconfigured our database to optimize these tables more frequently and in smaller batches. This keeps fragment counts low and ensures real-time queries remain fast and stable. Additionally, we have added monitoring and alerting on fragment count and size per table so we can detect abnormal accumulation early and prevent similar incidents in the future.

The issue causing delays in data ingestion has been fully resolved. Ingestion throughput has returned to normal operating levels, and all data is being processed as expected. There was no data loss during the incident. We will publish a post-mortem with additional details and preventive actions in a follow-up communication. Thank you for your patience and understanding.