pool.ntp.org Outage History

Started Sep 11, 2026, 10:00 PM UTC· resolved Sep 11, 2026, 10:00 PM UTC

1. resolved Sep 11, 2026, 10:00 PM UTC

   During the incident today we also noticed (belatedly) that the status page metrics updater had been failing because the status page API key had expired, oops.

Read the full incident report →

Started Sep 11, 2026, 07:28 PM UTC· resolved Sep 11, 2026, 09:51 PM UTC

1. investigating Sep 11, 2026, 07:28 PM UTC

   The database cluster is crash looping and/or starting up very slowly after a crash. We're investigating.

2. identified Sep 11, 2026, 07:49 PM UTC

   The symptoms and immediate cause have been identified and remedied. Working on restoring the database cluster to a healthy state.

3. resolved Sep 11, 2026, 09:51 PM UTC

   This has been resolved. The cluster is healthy again and the underlying causes have been resolved.

4. postmortem Sep 11, 2026, 10:01 PM UTC

   MySQL went down because our Ceph storage cluster filled up. That traces back to a Ceph security upgrade we'd just finished. The new version checks S3 upload signatures more strictly than before, and that exposed a bug in the S3 client our monitoring system uses: it doesn't sign one of the headers Ceph now requires. Its uploads got rejected, and instead of giving up it retried forever. Each upload is really several requests: the bulk data went through fine, but the small file that finalizes it kept getting rejected, so the data piled up without ever completing into anything usable. More than a terabyte of duplicate data accumulated in about half a day, more once you count replication overhead. A full Ceph cluster stops accepting writes from everything, and our database lives on the same cluster, so all three MySQL nodes crashed at the same time. Ceph was back as soon as we'd cleaned out the orphaned uploads. MySQL took longer: two of the three nodes had fallen too far behind to rejoin automatically, and their rejoin attempts took the primary down twice more before we fixed them by hand. The site was down a bit over two hours in total. Monitoring data from that window never got stored and can't be recovered. No other data was lost. The client library's bug is already fixed upstream, but the fix hasn't shipped in a release Mimir or Tempo use yet, so we moved that traffic to a connection where it signs everything correctly instead of waiting for it.

Read the full incident report →

Started Mar 02, 2026, 02:00 AM UTC· resolved Mar 02, 2026, 02:00 AM UTC

1. resolved Mar 02, 2026, 05:29 AM UTC

   The website (in particular the graphs) had a prolonged outage today. The graphs are back now, but will have some expected maintenance windows in the coming weeks related to the infrastructure migration (see https://news.ntppool.org/ )

Read the full incident report →

Started Sep 26, 2025, 12:52 AM UTC· resolved Apr 26, 2026, 08:05 AM UTC

1. investigating Sep 26, 2025, 12:52 AM UTC

   The newly reset database cluster (from last night) decided to blow itself up again. I think the extra query load and data from the new monitoring system is making the setup fragile. Investigating.

2. identified Sep 26, 2025, 01:25 AM UTC

   With a bit of help the kubernetes operator got the cluster working again, at least for now.

3. identified Sep 27, 2025, 09:40 PM UTC

   The database croaked again! I'm restoring from a ~20 minute old backup.

4. monitoring Sep 28, 2025, 01:37 AM UTC

   ok, we are stable again -- hopefully.

5. resolved Apr 26, 2026, 08:05 AM UTC

   System has been migrated to the new infrastructure.

Read the full incident report →

Started Sep 24, 2025, 10:44 PM UTC· resolved Sep 25, 2025, 10:26 AM UTC

1. investigating Sep 24, 2025, 10:44 PM UTC

   The database is having an outage (again!). Investigating.

2. monitoring Sep 25, 2025, 12:20 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 25, 2025, 10:26 AM UTC

   Database cluster has been re-installed and stable for about an hour now.

Read the full incident report →

Started Jul 05, 2025, 04:00 AM UTC· resolved Jul 05, 2025, 12:42 PM UTC

1. investigating Jul 05, 2025, 07:19 AM UTC

   The upgraded database is having a hard time, again!

2. monitoring Jul 05, 2025, 09:15 AM UTC

   It's been recovered (since about an hour ago). More information at https://community.ntppool.org

3. resolved Jul 05, 2025, 12:42 PM UTC

   The cluster is stable (and has been reconfigured with many more resources which seemed to be one of the causes of the frequent crashes and eventual loss of sync).

Read the full incident report →

Started Jul 02, 2025, 08:00 AM UTC· resolved Jul 02, 2025, 08:00 AM UTC

1. resolved Jul 02, 2025, 01:32 PM UTC

   The management website was partially offline for a database upgrade (that took longer than expected). Some data / updates from late Tuesday July 1st might have been lost.

Read the full incident report →

Started Apr 13, 2025, 12:45 AM UTC· resolved Apr 13, 2025, 01:51 AM UTC

1. investigating Apr 13, 2025, 12:45 AM UTC

   The kubernetes cluster is having a rough day again; management and and monitoring is down. DNS servers are operating normally.

2. investigating Apr 13, 2025, 12:45 AM UTC

   We are continuing to investigate this issue.

3. investigating Apr 13, 2025, 01:51 AM UTC

   All clear again (since a bit ago). Fixing a minor problem I caused a major problem by doing an unexpected update to a network configuration.

4. resolved Apr 13, 2025, 01:51 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 05, 2025, 02:19 AM UTC· resolved Apr 05, 2025, 08:34 PM UTC

1. identified Apr 05, 2025, 02:19 AM UTC

   A fiber cut is affecting the management system kubernetes cluster. The NTP Pool DNS service is unaffected. Equinix Metal status: https://status.equinixmetal.com/incidents/3svngx53jz9j

2. identified Apr 05, 2025, 03:21 PM UTC

   Fiber cut has been fixed and kubernetes cluster restored (over night).

3. resolved Apr 05, 2025, 08:34 PM UTC

   The system has been fully functional since the previous update; we've been monitoring since.

Read the full incident report →