pool.ntp.org incident

Monitoring and web system outage

Major Resolved View vendor source →

pool.ntp.org experienced a major incident on September 1, 2023 affecting Management Portal and Public website and 1 more component, lasting 4h 52m. The incident has been resolved; the full update timeline is below.

Started
Sep 01, 2023, 03:46 AM UTC
Resolved
Sep 01, 2023, 08:39 AM UTC
Duration
4h 52m
Detected by Pingoru
Sep 01, 2023, 03:46 AM UTC

Affected components

Management PortalPublic websiteDNS updates

Update timeline

  1. investigating Sep 01, 2023, 03:46 AM UTC

    Many containers across the central cluster restarted; monitoring and web services are unavailable or sporadically working. The DNS / NTP service is unaffected.

  2. identified Sep 01, 2023, 03:47 AM UTC

    We're rebooting some servers to get the Ceph cluster storage working again.

  3. monitoring Sep 01, 2023, 04:12 AM UTC

    Most or all services should be back up again.

  4. resolved Sep 01, 2023, 08:39 AM UTC

    This incident has been resolved.