RRPproxy incident

ns2.rrpproxy.net and ns2.dnsres.net affected by network outage

Major Resolved View vendor source →

RRPproxy experienced a major incident on March 10, 2021 affecting DNS resolution, lasting 5d 9h. The incident has been resolved; the full update timeline is below.

Started
Mar 10, 2021, 12:45 AM UTC
Resolved
Mar 15, 2021, 10:15 AM UTC
Duration
5d 9h
Detected by Pingoru
Mar 10, 2021, 12:45 AM UTC

Affected components

DNS resolution

Update timeline

  1. identified Mar 10, 2021, 01:36 AM UTC

    ns2.rrpproxy.net and ns2.dnsres.net are currently unavailable due to a large network outage in the backing data center.

  2. identified Mar 10, 2021, 01:49 AM UTC

    The data center team is already working on resolving the issues with all resources available.

  3. identified Mar 10, 2021, 02:53 AM UTC

    The data center team is still working on resolving the issues. Unfortunately we cannot provide any ETA at this point of time.

  4. identified Mar 10, 2021, 05:08 AM UTC

    Unfortunately the network reachability of both ns2.rrpproxy.net and ns2.dnsres.net is affected by a major incident in one data center location. We are in contact with the responsible network team of the data center to get the traffic for these systems rerouted to a different data center location.

  5. identified Mar 10, 2021, 09:29 AM UTC

    Update: Our Web Forwarding is affected as well and currently unavailable. We continue to work on fixing these issues.

  6. identified Mar 10, 2021, 12:18 PM UTC

    Update: As a workaround we're switching our Nameserver IP addresses for ns2.rrpproxy.net and ns2.dnsres.net as follows: ns2.rrpproxy.net from: IPv4 178.33.34.29 IPv6 2001:41d0:d:7ea:178:33:34:29 to: IPv4 51.68.180.207 IPv6 2001:41d0:700:1bcf:51:68:180:207 ns2.dnsres.net from: IPv4 188.165.164.171 IPv6 2001:41d0:d:7ea:188:165:164:171 to: IPv4 51.75.87.7 IPv6 2001:41d0:700:1bcf:51:75:87:7 Please regard that depending on TTL setting and caching it may take a while until new NS IP addresses are populated.

  7. identified Mar 10, 2021, 01:36 PM UTC

    Update on NS: ns2.rrpproxy.net came back online at their location in the data center having an incident. As a consequence, they answer on queries. However, we will keep the hostname switched to the IP address communicated in the previous update and will switch back when we're confident the issues at the data center have been resolved. EDIT: removed information about ns2.dnsres.net; only ns2.rrpproxy.net came back online

  8. identified Mar 10, 2021, 03:24 PM UTC

    Update: ns2.dnsres.net came back online at its location in the data center having an incident. As a consequence, it answers on queries. However, we will keep the hostname switched to the IP address communicated in the previous update and will switch back when we're confident the issues at the data center have been resolved.

  9. identified Mar 10, 2021, 05:31 PM UTC

    Update on Web Forwarding: We deployed a new Web Forwarding set and are currently adjusting domains. The new IP addresses are: IPv4 old: 51.91.60.4 --> new: 66.206.3.38 IPv6 old: 2001:41d0:301:100:51:91:60:4 --> new: 2604:4500:c:4:66:206:3:38 The scripts to adjust Web Forwarding for affected domains are running. We expect to have all domains adjusted in a few hours.

  10. identified Mar 11, 2021, 09:41 AM UTC

    Short update: We're still working on bringing back systems to normal state.In particular, we're busy setting up new nodes for ns2.rrpproxy.net and ns2.dnsres.net and will communicate new IP addresses as soon as they are available. Until then the communicated IP addresses for our NS serve as replacement. For Web Forwarding scripts are still running to update affected domains.

  11. identified Mar 12, 2021, 09:14 AM UTC

    Update on Web Forwarding: Meanwhile all Domains using Web Forwarding have been adjusted and are now using our new IP addresses: IPv4 66.206.3.38 IPv6 2604:4500:c:4:66:206:3:38

  12. monitoring Mar 12, 2021, 11:38 AM UTC

    Update on Nameservers: We finished setting up new IP addresses at their new permanent location for ns2.rrpproxy.net and ns2.dnsres.net: ns2.rrpproxy.net: IPv4 66.206.3.124 / IPv6 2604:4500:c:3:66:206:3:124 ns2.dnsres.net: IPv4 66.206.3.126 / IPv6 2604:4500:c:3:66:206:3:126 Also, we finished setting up a new IP address for our nameserver providing DNS Zone Xfer IN/OUT at its new permanent location: ns2-eu.key-systems.net: IPv4 66.206.25.194 / IPv6 2604:4500:c:3:66:206:25:194 Note: Please make sure this IP address can connect to your DNS server (Master, Hidden Master or Slave) for outgoing/incoming DNS Zone transfers (notify/AXFR). I.e. respective firewall rules need to be set up. All IP addresses are live as of now. Please be aware that depending on TTL setting and caching it may take a while until these new NS IP addresses are populated. Documentation will be adjusted once all work has been finalised.

  13. resolved Mar 15, 2021, 10:15 AM UTC

    Update As already posted in our last update we have set up new IP addresses replacing the systems that were affected by the incident in the data center last Wednesday, March 10. The new systems are up and running at their new permanent location. We continue to monitor the situation. For future adjustments we will inform appropriately.