Empty RPKI repositories (and missing trust anchor certificate) published

Update timeline

1. identified Jan 07, 2026, 03:27 PM UTC

   During a short time window (15:11 - 15:20 UTC) our RPKI systems published an empty set of data to the publication point. This propagated to both the rsync and RRDP endpoints. We are investigating the situation and provide more information later.

2. monitoring Jan 07, 2026, 03:28 PM UTC

   The initial incident has been resolved and we are monitoring the situation.

3. monitoring Jan 07, 2026, 03:42 PM UTC

   This issues was triggered by a software release that contained a database change. This change caused our our systems to clear and re-publish the complete repository.

4. resolved Jan 07, 2026, 04:11 PM UTC

   This incident has been resolved. We'll follow up tomorrow with a RFO.

5. postmortem Jan 08, 2026, 03:28 PM UTC

   On 7 January 2026 at 15:11 UTC, all objects were removed from both our RRDP and rsync RPKI repositories. At 15:20 UTC, the repository state was restored, and all objects were back. At 15:08 UTC, we did a production deployment of our CA software on the first application node. We deploy one node at a time so that we have zero downtime during deployments. We carried out a database DDL change on the table storing published objects, which took roughly 40 seconds to execute. During those 40 seconds, the online node queried the published objects in order to publish them. This transaction had to wait for the table lock held by the DDL change to be released. When the lock was released, PostgreSQL returned zero rows. We learned that this is a known caveat \[1\]: after rewrite commits, the table will appear empty to concurrent transactions. We will implement circuit breakers between all phases of our publication process to prevent this situation from happening in the future. \[1\] - [https://www.postgresql.org/docs/18/mvcc-caveats.html](https://www.postgresql.org/docs/18/mvcc-caveats.html) Apologies for the inconvenience this has caused.