Hex experienced a critical incident on November 12, 2024 affecting Main site, lasting 2h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 12, 2024, 04:18 PM UTC
We are currently investigating this issue.
- investigating Nov 12, 2024, 04:18 PM UTC
We are continuing to investigate this issue.
- investigating Nov 12, 2024, 05:17 PM UTC
We are continuing to investigate this issue.
- monitoring Nov 12, 2024, 05:38 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 12, 2024, 07:08 PM UTC
On November 12, 2024 from 8:06 AM to 10:21 PM PT, Hex customers experienced platform instability, including issues with kernel execution and site access. This disruption stemmed from an unintended effect of a recent database migration, which ultimately caused a backlog and impacted overall platform performance. Our engineering team promptly identified the root cause and took corrective actions by resetting kernels to alleviate database congestion. Once the database stabilized, we rolled back the migration change, fully restoring service by 10:21 PM PT. To prevent similar issues in the future, we are enhancing our automated tests to better validate database migrations before deployment, ensuring they don’t inadvertently impact platform reliability. Thank you for your patience and understanding.