ARIN experienced a minor incident on August 11, 2022 affecting RPKI RRDP Repository, lasting 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 11, 2022, 04:20 PM UTC
We are currently investigating an issue with RPKI RRDP.
- investigating Aug 11, 2022, 04:49 PM UTC
This incident has been resolved.
- resolved Aug 11, 2022, 04:50 PM UTC
This incident has been resolved.
- postmortem Aug 11, 2022, 09:42 PM UTC
At 11:20 EDT, a configuration management change installed mismatched CA certificate/keys on a subset of nodes that serve the RPKI RRDP repository. This triggered alarms indicating degraded performance of the RPKI RRDP services. The repository generation was subsequently paused while the misconfigured nodes were being identified and removed from our DNS rotation. New certs and keys were pushed to the impacted systems and they were returned to the DNS rotation. At 12:50 the repository generation was restarted and full functionality of the RPKI RRDP Services was restored. RPKI RSYNC services were functional throughout the incident, but publication of ROAs would have been delayed during the incident. Processes and procedures have been updated to prevent a future reoccurrence of this type of issue.