RIPE Network Coordination Centre incident
RIS: RRC12 MRT update and bview files are not produced
Resolved after 3 days 23 hours, affecting RIS (Routing Information Service), on October 2, 2026.
This incident has been resolved. After the initial process recovered, the bview and update files since 2 October were produced successfully. The incident trigger was a high volume of updates from one peer. This caused load in a single kafka partition. When that process restarted, it hit a latent bug where it tried to re-scan a large volume of data on startup. This scan repeatedly timed out, which prevented the process from making progress. Additionally, due to a misconfiguration in the code, this scan took an order of magnitude longer than it should. After fixing this configuration issue the process recovered. We will work on a structural fix for the latent bug in the next sprint.
Affected components
Update timeline
- monitoring Oct 02, 2026, 07:42 AM UTC
Oct 2, 09:42 CEST Investigating - The latest available MRT update file for RRC12 has a timestamp of at 01:00 UTC, October 2nd, 2026. We are investigating.
- investigating Oct 02, 2026, 07:42 AM UTC
The latest available MRT update file for RRC12 has a timestamp of 01:00 UTC, October 2nd, 2026. The latest available BVIEW file for RRC12 has a timestamp of 00:00 UTC , October 2nd, 2026. We are investigating.
- identified Oct 02, 2026, 11:26 AM UTC
At 1.30UTC, on October 2nd, 2026, a DE-CIX peer started sending an unusually high number of BGP messages to RRC12. Our processing pipeline consuming from kafka did not keep up with this volume of updates, causing a process to time out. When this process restarts, it times out again before making enough progress. We are now observing an impact on the production of other RRCs updates files, that are being delayed. We are troubleshooting this issue.
- identified Oct 05, 2026, 12:46 PM UTC
We continue to work on this issue. We have identified the root cause of the pipeline stall and are developing a workaround.
- resolved Oct 06, 2026, 06:44 AM UTC
This incident has been resolved. After the initial process recovered, the bview and update files since 2 October were produced successfully. The incident trigger was a high volume of updates from one peer. This caused load in a single kafka partition. When that process restarted, it hit a latent bug where it tried to re-scan a large volume of data on startup. This scan repeatedly timed out, which prevented the process from making progress. Additionally, due to a misconfiguration in the code, this scan took an order of magnitude longer than it should. After fixing this configuration issue the process recovered. We will work on a structural fix for the latent bug in the next sprint.
Your product runs on more than RIPE Network Coordination Centre
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services