Higher Logic experienced a major incident on January 26, 2025, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jan 26, 2025, 03:23 AM UTC
Some sites were showing a server error when trying to access
- postmortem Jan 30, 2025, 02:01 PM UTC
**What Happened** On Saturday nights, we run a full reindex operation on each database before the Full backups run against our databases. During the reindex operation, our application servers started having issues connecting to the databases. **Root Cause** A recent change in the application stack caused a significantly higher rate of log entries to a system Log table. Due to the increased size of this table, the reindex process locked that table for a significant amount of time. In turn, this caused issues for all OC application servers connecting to the databases. **Corrective Actions** * A code change has been released to production to fix the uncontrolled growth of the log table. * Enhance monitors that will alert when unexpected growth conditions happen on key system Log tables.