Higher Logic experienced a critical incident on March 27, 2025 affecting Community, lasting 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 27, 2025, 05:23 PM UTC
We are currently investigating this issue.
- monitoring Mar 27, 2025, 05:35 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 27, 2025, 06:24 PM UTC
This incident has been resolved.
- postmortem Apr 25, 2025, 01:53 PM UTC
### What Happened: On Wednesday morning all our Online Community sites started returning 504 Gateway Timeout and 503 service unavailable errors. This also affected Informz and Real Magnet admin portals. ### Root Cause: A backend in the Inside Proxies configuration was not responding, making the proxy config invalid. This caused the proxy servers to error on startup. They continued to cycle until no inside proxy servers were available. ### Details: We determined that some clean-up work in DNS removed an A record for the backend because it was pointing to a non-existent private IP. Without this A record present, the proxy config considered this an invalid backend. This invalid backend caused startup errors to occur in all our inside proxy servers resulting in 504 Gateway Timeout and 503 service unavailable errors. ### Corrective Actions: 1. The invalid backend entry has been removed from the proxy config. 1. A ticket will be created to review all our proxy configurations to ensure all backends are valid and have rules. If not, they will be safely removed, along with the corresponding A record in DNS