RRPproxy incident

DNS resolution issues in US and Ireland regions

Major Resolved View vendor source →

RRPproxy experienced a major incident on May 6, 2021 affecting DNS resolution, lasting 2h 24m. The incident has been resolved; the full update timeline is below.

Started
May 06, 2021, 07:38 PM UTC
Resolved
May 06, 2021, 10:03 PM UTC
Duration
2h 24m
Detected by Pingoru
May 06, 2021, 07:38 PM UTC

Affected components

DNS resolution

Update timeline

  1. investigating May 06, 2021, 07:38 PM UTC

    We are currently investigating this issue.

  2. identified May 06, 2021, 07:43 PM UTC

    We are affected by regional issues (US & Ireland) of UltraDNS, which provide DNS services for rrpproxy.net. This also affects resolving ns*.rrpproxy.net, as well as customers using the premium DNS service.

  3. identified May 06, 2021, 08:19 PM UTC

    Received statement from UltraDNS: --- We are experiencing a service degradation in authoritative DNS resolution with greatest impact to queries routed to US East for resolution and also impact to EU West regions. [...] Current Status: We've restored service to some regions but are continuing to see impact at this time. Our teams are continuing to work to restore service to normal levels. ---

  4. monitoring May 06, 2021, 08:26 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved May 06, 2021, 10:03 PM UTC

    This incident has been resolved.

  6. postmortem May 07, 2021, 01:02 PM UTC

    # Statement for DNS resolution issues on 2021-05-06 ## Description UltraDNS, our DNS provider for the domain [rrpproxy.net](http://rrpproxy.net), had a service degradation in authoritative DNS resolution with greatest impact to queries routed to US East for resolution and also impact to EU West regions on 2021-05-06 between 18:00 UTC and 21:00 UTC. As a consequence resolution for hostnames underneath [rrpproxy.net](http://rrpproxy.net) was only partially available during this timeframe. This had direct consequences on the reachability of our services, including website, web interface, API gateways and nameservers. According to UltraDNS by 20:00 UTC most services were restored and their Engineering sent the all clear around 21:00 UTC. We were able to confirm that domains using UltraDNS were reachable again. ## Impact on Security No data has been corrupted. ## Follow Up Actions Will be discussed internally if changes to name servers for [rrpproxy.net](http://rrpproxy.net) will be applied.