pool.ntp.org incident

Excessive load on NTP servers

Minor Resolved View vendor source →

pool.ntp.org experienced a minor incident on December 19, 2016, lasting 11d 19h. The incident has been resolved; the full update timeline is below.

Started
Dec 19, 2016, 02:16 AM UTC
Resolved
Dec 30, 2016, 09:22 PM UTC
Duration
11d 19h
Detected by Pingoru
Dec 19, 2016, 02:16 AM UTC

Update timeline

  1. investigating Dec 19, 2016, 02:16 AM UTC

    Since around last Tuesday the NTP servers in some countries (or so it appears) have seen massively increased load. We believe it's a vendor not using the vendor system and haven't yet been able to identify who it is. It seems to mostly be affecting the United States, United Kingdom, Australia and New Zealand. If you are running an NTP server, your service is greatly appreciated. If your server is seeing more queries than you can handle, please consider lowering the "net speed" in the server preferences rather than removing the server altogether. If your server can handle more NTP queries than it currently gets, please consider increasing the same setting.

  2. identified Dec 20, 2016, 02:09 AM UTC

    The trouble in some regions of the world appears to be caused by the Snapchat application: http://mailman.nanog.org/pipermail/nanog/2016-December/089588.html We've tried reaching out to them.

  3. identified Dec 20, 2016, 03:18 AM UTC

    More information: https://news.ntppool.org/2016/12/load/ https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18

  4. monitoring Dec 21, 2016, 07:57 PM UTC

    Snapchat released an update to their app yesterday with a fix. They've also added some high capacity servers in some of the affected regions. 👍

  5. resolved Dec 30, 2016, 09:22 PM UTC

    The load appears to be just about back to normal. By server count we've almost mostly recovered from the "lost" servers during the event. https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18/10