Hex had 27 outages in the last 2 years totaling 596h 44m of downtime — averaging 1.1 incidents per month.
There were 27 Hex outages since June 20, 2024 totaling 596h 44m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We have identified the root cause upstream of Hex with some VPN and web firewall providers. We are working on a resolution with these providers. In the meantime, we recommend users disconnect from their VPN and/or disable firewalls when accessing Hex if possible.
We have identified the root cause upstream of Hex with some VPN and web firewall providers. We have been working with these providers on a resolution and are monitoring results. We expect some users will now be able to use their VPN and/or firewall with no impact to any Hex projects, but you may need need to restart your browser before accessing the impacted projects. If you are continuing to experience issues with projects containing HTML, please reach out to [email protected] with detail on the VPN and/or firewall provider you are using.
As noted yesterday, we have identified the root cause upstream of Hex with some VPN and web firewall providers. We have been working with these providers on a resolution and are continuing to monitor results. Depending on your VPN and/or firewall provider, issues with HTML outputs may be fully resolved for your organization, but you may need need to restart your browser. If you are continuing to experience issues with projects containing HTML, please reach out to [email protected] with detail on the VPN and/or firewall provider you are using.
We have been working with VPN and firewall providers to fully resolve this issue for all Hex users. While most users are no longer experiencing issues, please reach out to [email protected] with detail on your VPN and/or firewall provider if you believe you may still be impacted. We anticipate all impacts will be fully resolved within the next week.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue using 3.11 custom images. It is expected that projects using customer images are able to acquire a kernel, but are not able to query data sources.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
This incident has been resolved.
AWS is experiencing issues in the us-east-1 region that may affect some Hex services for a subset of customers. We are closely monitoring AWS's updates: https://health.aws.amazon.com/health/status
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
A fix has been implemented and systems are now operation, we will continue to monitor.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue acquiring kernels for some customers.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We've identified an issue with custom kernels failing to build and are actively working on a fix.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We've identified an issue with published apps using the pre-run cells in the background feature and are working on implementing a fix.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
When users copy to clipboard and/or download CSV they may see HTML rather than their data. The issue has been identified and a fix is being implemented. In the meantime, a refresh of the page should resolve the issue.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Slack is currently experiencing an outage which impacts the ability for Hex to send notifications to this channel. We are monitoring this outage at https://slack-status.com/ and encourage you to do the same if you rely on this functionality.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
On November 12, 2024 from 8:06 AM to 10:21 PM PT, Hex customers experienced platform instability, including issues with kernel execution and site access. This disruption stemmed from an unintended effect of a recent database migration, which ultimately caused a backlog and impacted overall platform performance. Our engineering team promptly identified the root cause and took corrective actions by resetting kernels to alleviate database congestion. Once the database stabilized, we rolled back the migration change, fully restoring service by 10:21 PM PT. To prevent similar issues in the future, we are enhancing our automated tests to better validate database migrations before deployment, ensuring they don’t inadvertently impact platform reliability. Thank you for your patience and understanding.
We are currently investigating this issue.
Hex is experiencing degraded performance; some users are unable to acquire kernels and run projects. We are actively investigating the root cause and will update as soon as possible on a resolution.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.