RIPE Network Coordination Centre incident

Issues reaching RIPEstat

Major Resolved View vendor source →

RIPE Network Coordination Centre experienced a major incident on August 29, 2025 affecting RIPEstat, lasting 1h 34m. The incident has been resolved; the full update timeline is below.

Started
Aug 29, 2025, 01:48 PM UTC
Resolved
Aug 29, 2025, 03:22 PM UTC
Duration
1h 34m
Detected by Pingoru
Aug 29, 2025, 01:48 PM UTC

Affected components

RIPEstat

Update timeline

  1. investigating Aug 29, 2025, 01:48 PM UTC

    We are currently investigating this issue.

  2. monitoring Aug 29, 2025, 02:45 PM UTC

    Around 12:35-12:40 UTC there was a severe increase in load on the RIPEstat application servers. This caused the application servers to start swapping, and performance to degrade. Between 14:00-14:05 requests started to fail. The root cause seems to be a shift in request pattern. The situation was complicated by a change in configuration management that meant we could not restart our application processes in big batches, but had to do this machine-by-machine. As an intervention we will limit swap usage by our python processes to make failures (and recovery from failures) faster.

  3. resolved Aug 29, 2025, 03:22 PM UTC

    This issue has been resolved. We will work on the proposed mitigations soon.