Hypernode had 27 outages in the last 2 years totaling 3h 10m of downtime — averaging 1.1 incidents per month.
There were 27 Hypernode outages since September 10, 2024 totaling 3h 10m of downtime. Each is summarised below — incident details, duration, and resolution information.
Some of our customers are experiencing DNS related problems with their Hypernode server. The problem has been identified, our team is implementing a solution to the problem.
A fix has been implemented, we are monitoring the problem.
This incident has been resolved.
This morning we performed maintenance on our internal infrastructure, in preparation for a future migration. Shortly afterwards our support team started receiving a large amount of error reports related to DNS, and we performed a rollback at 10:38 this morning. Throughout the rest of the day we still received evidence that a small amount of visitors, with specific internet providers, were having problems resolving certain DNS records.
After extensive research and testing, we learned that our DNS provider performed maintenance this morning too, at the same time as our internal maintenance. During this maintenance they shut-off a DNS server that had been decommissioned 4 weeks ago. In preparation of this, we had changed our DNS settings 4 weeks ago to indicate that this old server should not longer be used. Unfortunately, it seems that certain servers from these ISP's are not respecting the 1 day validity (TTL / Time-to-live) on these DNS records, and were still contacting this old server, even though we indicated not to use those servers anymore 4 weeks ago. We worked with our DNS provider and the decommissioned server has been restored for now. We are working on contacting the ISP's that are not following the internet standards for DNS TTL, in order to prevent this issue from returning at a later date.