Higher Logic experienced a critical incident on April 23, 2025 affecting Community and Marketing Enterprise (Real Magnet) and 1 more component, lasting 1h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 23, 2025, 11:49 AM UTC
We are currently investigating this issue.
- monitoring Apr 23, 2025, 12:41 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 23, 2025, 01:27 PM UTC
This incident has been resolved.
- postmortem Apr 25, 2025, 05:42 PM UTC
### What Happened: On Wednesday morning all our Online Community sites started returning 504 Gateway Timeout and 503 service unavailable errors. This also affected Informz and Real Magnet admin portals. ### Root Cause: A backend in the Inside Proxies configuration was not responding, making the proxy config invalid. This caused the proxy servers to error on startup. They continued to cycle until no inside proxy servers were available. ### Details: We determined that some clean-up work in DNS removed an A record for the backend because it was pointing to a non-existent private IP. Without this A record present, the proxy config considered this an invalid backend. This invalid backend caused startup errors to occur in all our inside proxy servers resulting in 504 Gateway Timeout and 503 service unavailable errors. ### Corrective Actions: 1. The invalid backend entry has been removed from the proxy config. 2. A ticket will be created to review all our proxy configurations to ensure all backends are valid and have rules. If not, they will be safely removed, along with the corresponding A record in DNS