ARIN incident

Inability to fully serve RPKI traffic during code deploy

Minor Resolved View vendor source →

ARIN experienced a minor incident on January 23, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Jan 23, 2023, 08:08 PM UTC
Resolved
Jan 03, 2023, 06:00 AM UTC
Duration
—
Detected by Pingoru
Jan 23, 2023, 08:08 PM UTC

Update timeline

  1. resolved Jan 23, 2023, 08:08 PM UTC

    During code deployment (migration to new repository generation) we were unable to fully serve RPKI traffic.

  2. postmortem Jan 23, 2023, 08:08 PM UTC

    During the deployment of new software in the RPKI repository generation systems it was necessary to roll the RRDP/session\_id. This is an operation that is well-defined by the RPKI standards and well-supported by the RPKI validators, but it causes clients to download a full snapshot rather than a delta file. The subsequent load of snapshot requests resulted in slow or incomplete downloads of the repository. The system remained in this impaired state about 60 minutes. During this time validators would continue serving from their cache without interruption. Validators without a cached copy of the ARIN repository were not capable of responding with cached data. Additional compute capacity was added to the RPKI infrastructure, and subsequent downloads started completing successfully and load returned to normal levels.