High error rate and degrated performance for RIPEstat

Update timeline

1. monitoring Jun 18, 2026, 11:45 AM UTC

   Jun 18, 13:45 CEST Investigating - Since around 10:15 UTC, RIPEstat has been experiencing elevated error rates and increased latency. The issue is caused by a failing node in HBase, the distributed database used by many RIPEstat datasets. We are currently investigating the issue.

2. investigating Jun 18, 2026, 11:45 AM UTC

   Since around 10:15 UTC, RIPEstat has been experiencing elevated error rates and increased latency. The issue is caused by a failing node in HBase, the distributed database used by many RIPEstat datasets. We are currently investigating the issue.

3. identified Jun 18, 2026, 12:20 PM UTC

   We have identified a faulty node in the HBase cluster and applied mitigations. The error rate remains elevated at around 5%, but RIPEstat functionality has improved. We are continuing to monitor the situation.

4. monitoring Jun 18, 2026, 01:51 PM UTC

   The root cause likely is a cascade of HBase region server restarts after a Zookeeper session expired due to an unknown cause. The most likely cause is timeout after a worker did not communicate for 40 seconds. We will add high resolution network monitoring between each zookeeper node and all zookeeper clients to track potential network issues.

5. resolved Jun 22, 2026, 07:08 AM UTC

   This incident has been resolved.