NoIP incident

Delays Processing Dynamic DNS Updates

Minor Resolved View vendor source →

NoIP experienced a minor incident on July 3, 2023 affecting Free Dynamic DNS and Enhanced Dynamic DNS and 1 more component, lasting 13h 15m. The incident has been resolved; the full update timeline is below.

Started
Jul 03, 2023, 01:37 PM UTC
Resolved
Jul 04, 2023, 02:53 AM UTC
Duration
13h 15m
Detected by Pingoru
Jul 03, 2023, 01:37 PM UTC

Affected components

Free Dynamic DNSEnhanced Dynamic DNSPlus Managed DNSDynamic Update API

Update timeline

  1. investigating Jul 03, 2023, 01:37 PM UTC

    We are investigating reports of delays in processing Dynamic DNS updates causing some Dynamic DNS names to resolve to previous IP addresses. We will provide an update once we have more information.

  2. investigating Jul 03, 2023, 04:46 PM UTC

    We are working diligently to process the backlog of Dynamic DNS updates. As we continue to mitigate the issue, we'll provide more updates as they become available.

  3. identified Jul 03, 2023, 08:20 PM UTC

    Update: We are continuing our mitigation efforts and have reduced the backlog of Dynamic DNS updates by 90%, however, we are still experiencing delays in processing inbound update requests. We will provide another update as our mitigation efforts progress.

  4. monitoring Jul 03, 2023, 09:55 PM UTC

    We have resolved the issue that caused saturation of our Free and Enhanced Dynamic DNS update queue. At this time, all new updates are being processed in real-time. Updates for Plus Managed DNS service were not affected. Between 11:45 PM Pacific on 7/2/2023 and 2:00 PM Pacific on 7/3/2023, Dynamic DNS updates were received by our servers, however, our queue that pushes these updates to DNS fell behind. After several attempts to get the queue back in sync, It became apparent that in order to recover the queue we would have to delete records from the queue. Now that the queue has recovered, the majority of Dynamic DNS updates that did not make it to DNS will self-recover when a dynamic update client sends its next update. If your hostname is out of sync with what is displayed on the NoIP.com website compared to DNS you can also manually submit an update. To do so login to your No-IP account go to “My Services”, “DNS Records” find the hostname out of sync, click “Modify” then “Update Hostname”. Once updated you should see the hostname resolving to the correct target within 60 seconds. If you do not, you may need to flush your DNS recursive cache. We’re working to validate that all updates during the event are processed to ensure we’re resolving to the most recent DNS update that we received. However, if you need your hostname to be updated immediately please update your hostname through our website, NoIP.com. We sincerely apologize for the impact this may have caused you.

  5. resolved Jul 04, 2023, 02:53 AM UTC

    We have observed stabilization processing Dynamic DNS updates from our queue. All systems are operational at this time.