Higher Logic experienced a major incident on July 8, 2025 affecting Community, lasting 3h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 08, 2025, 11:10 AM UTC
We are currently investigating this issue.
- monitoring Jul 08, 2025, 12:23 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 08, 2025, 02:11 PM UTC
This incident has been resolved.
- postmortem Jul 11, 2025, 05:09 PM UTC
### Summary On July 8, 2025, Higher Logic Online Communities sites experienced a service disruption that made them temporarily inaccessible. Our engineering teams identified and resolved the issue, restoring full service. We sincerely apologize for the impact this incident had on our customers and their communities. This report outlines the event, its cause, and the actions we are taking to improve our systems and prevent recurrence. Impact Between approximately 7:10 AM and 8:23 AM EDT, Higher Logic Online Communities sites were inaccessible. Our team restored service and, after a period of monitoring, confirmed all systems were fully operational at 10:11 AM EDT. ### Root Cause Analysis The service disruption was triggered by the deployment of an incorrect configuration file to our production environment. During a routine update, a configuration file was deployed that contained a duplicated parameter. Our application's startup process includes a validation check to ensure all configuration data is correct. This check functioned as designed and identified the duplicated parameter as an invalid entry. As a safety measure, this validation failure halted the application's startup sequence to prevent potential data corruption or further instability, which resulted in the service becoming unavailable. While we have automated checks in place to prevent malformed files from being deployed, the check in place at the time did not test for this specific type of configuration error. ### Resolution Our engineering team resolved the incident by removing the invalid configuration file and reverting to the last known good version. This allowed the application to pass its startup validation and successfully restart, restoring service to all customers. ### Corrective and Preventative Actions We are committed to the stability of our platform and are taking the following steps to prevent this type of incident from happening again: **Enhanced Pre-Deployment Validation**: We have prioritized a critical update to our deployment process. This enhancement will ensure our automated pre-deployment checks perform the same validation that occurs when the application starts. This will allow us to automatically detect and block this type of configuration error before it can ever impact the production environment. We thank you for your patience and partnership. We are dedicated to learning from this event and continuously improving the reliability of our services