ARIN experienced a major incident on November 13, 2024 affecting RPKI RRDP Repository and RPKI Rsync Repository, lasting 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 13, 2024, 07:19 PM UTC
We are experiencing degraded performance on RPKI repositories for RRDP/rsync
- investigating Nov 13, 2024, 07:27 PM UTC
We are experiencing an outage of RPKI repositories.
- resolved Nov 13, 2024, 08:20 PM UTC
This incident has been resolved.
- postmortem Nov 13, 2024, 09:52 PM UTC
ARIN completed a scheduled, simulated failure of the ARIN RPKI repository today at 1500 EST to assess failure response under production conditions. At 13:05 EST \(UTC-5\) , ARIN began restricting RRDP access to the sites that serve the ARIN Resource Public Key Infrastructure \(RPKI\) Hosted and RPS \(hybrid\) repositories. At 14:15, all access was administratively blocked, and the repositories was made unavailable. At 15:05, access to the repositories was restored. Full redundancy of services was confirmed at 15:18. This decision to conduct the test in ARIN’s production environment was carefully considered. Testing outside of production would not have provided an accurate representation of performance and preparedness for repository failure conditions. Therefore, ARIN chose to simulate this scenario in a live environment to best evaluate real-world impacts and response. We appreciate the patience and cooperation of the RPKI community throughout this exercise, and conduct this testing is part of our commitment to high-quality RPKI services. Regards, Brad Gorman Senior Product Owner, Routing Security American Registry for Internet Numbers \(ARIN\)