Hypernode experienced a notice incident on September 10, 2024 affecting Hypernode Servers, lasting 2h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Sep 10, 2024, 10:47 AM UTC
This morning we performed maintenance on our internal infrastructure, in preparation for a future migration. Shortly afterwards our support team started receiving a large amount of error reports related to DNS, and we performed a rollback at 10:38 this morning. Throughout the rest of the day we still received evidence that a small amount of visitors, with specific internet providers, were having problems resolving certain DNS records.
- resolved Sep 10, 2024, 01:22 PM UTC
After extensive research and testing, we learned that our DNS provider performed maintenance this morning too, at the same time as our internal maintenance. During this maintenance they shut-off a DNS server that had been decommissioned 4 weeks ago. In preparation of this, we had changed our DNS settings 4 weeks ago to indicate that this old server should not longer be used. Unfortunately, it seems that certain servers from these ISP's are not respecting the 1 day validity (TTL / Time-to-live) on these DNS records, and were still contacting this old server, even though we indicated not to use those servers anymore 4 weeks ago. We worked with our DNS provider and the decommissioned server has been restored for now. We are working on contacting the ISP's that are not following the internet standards for DNS TTL, in order to prevent this issue from returning at a later date.