Is dataforest down?

The subnet 104.167.24.0/24 is unreachable since 18:57 due to a carpet bomb DDoS attack. We are working on the issue.

45.145.40.0/24 is unreachable since 19:10, too. Connectivity for 104.167.24.0/24 has been restored at 19:09.

45.145.40.0/24 is back online since 19:13, too. We have applied first measures to reduce the impact of the still running attacks and will apply additional traffic engineering later to ensure an equal traffic distribution across our POPs. Our backbone capacity is sufficient to mitigate these attacks, but traffic distribution needs to be optimized.

Reason for outage: After the recent edge router replacement at DRT-FRA8, some attack traffic was not distributed equally across both POPs for various reasons, and blackholing / prefix withdrawal was triggered for the mentioned prefixes. Our fixes applied shortly afterward have proven effective, and no further disruptions occurred despite ongoing attacks.

Between 20:18 and 20:23, we observed two brief periods of packet loss at one of our two PoPs, which affected some connections. Everything has been stable since then, and we're investigating the cause.

An issue at our EQX-FR4 PoP caused brief packet loss between 20:17:30 and 20:22:30 CEST. Traffic handled through our DRT-FRA8 PoP continued to operate normally throughout the incident, meaning a large part of our overall traffic was not affected. Across our network, total traffic dropped by around 40% for approximately one minute at the beginning of the incident. During recovery, we observed another much shorter drop of around 40% between 20:20:26 and 20:20:38. Within the following two minutes, traffic returned to normal levels. The issue was alerted internally at 20:22, followed by an incident call shortly after. During the investigation, we identified a likely root cause and implemented a first fix. The situation has remained stable since then.

We have validated our findings, and the situation has remained fully stable.

At 16:25, 16:32, and 16:35, a particular DDoS attack overloaded the DDoS filters at both POPs, causing temporary packet loss. A temporary fix has been implemented, and we are working on a permanent solution. Unprotected traffic was unaffected.

We have updated the initial version of this post, as we originally assumed that only one of our POPs was affected. In fact, both were.

The issue recurred at 20:39 for a few seconds, but with less impact as only one of our POPs was affected. We are still working on a permanent solution; our temporary fix has already reduced the impact.

At the previously specified times, an incorrect handling of dynamic attack pre-filtering during specific, very large DDoS attacks overwhelmed some of our filtering appliances, causing the issues described in this post. A permanent fix was deployed at 21:00, which has since been successfully verified under real-world traffic conditions. We can confirm that no further issues occurred other than those detailed here.

Two of our Avoro VM hosts, epyc7513-1 and epyc9374f-8, are currently experiencing an outage. The majority of Avoro VMs are not affected by the host outage itself. Customers can check the customer area to see whether any of their VMs are running on the affected hosts. VM control functions such as starting, stopping, and reinstalling are currently unavailable for all Avoro customers, as this functionality depends on the affected hosts. Affected VMs will be restarted on replacement hardware as soon as possible. We expect this process to take up to two hours. Individual VMs may be restored significantly earlier, as they will be brought back online sequentially. We will share further details as soon as they become available.

All affected VMs have been back online on replacement hardware for the past 10 minutes. Restoration was completed in well under 30 minutes, significantly faster than the previously estimated recovery time of up to two hours. We are now working to restore the control panel functionality.

Basic control panel functionality was restored at 18:50, including starting and stopping VMs as well as booting from mounted ISOs. Creating and restoring backups is also possible again. We are continuing to work on restoring VNC access and the reinstallation feature.

VNC access was restored at 22:37, followed by the reinstallation feature at 01:00. All previously affected control panel functionality has now been restored. Thank you for bearing with us while we worked to restore these services.

Following the incident on Tuesday, September 1, 2026, we want to provide a comprehensive postmortem regarding the root cause, our immediate response, and the resulting infrastructure upgrades. ### Incident Discovery & Scope On Tuesday morning, we detected unauthorized access to 2 of our approximately 50 Avoro virtualization hosts \(epyc7513-1 and epyc9374f-8\). Because this intrusion exploited a design flaw unique to this specific legacy setup, all other products \(such as PHP-Friends vServer, the dataforest cloud, and our internal systems\) operating in completely separate clusters were never at risk. By analyzing network traffic externally – completely outside the compromised systems – and monitoring active processes on the hosts, we confirmed the intrusion was strictly a known crypto-mining script. It exploited a Proxmox zero-day vulnerability disclosed just 90 minutes prior. We verified there was no ransomware, and no customer data or disk images were accessed or exfiltrated. ### Immediate Response & Control Panel Disconnect The Avoro customer area \(our control panel based on WHMCS\) historically required external communication with the cluster. For failover purposes, exactly the two affected nodes were equipped with a public IPv4 address to handle this traffic. Once we identified the intrusion, we completely locked down the Proxmox management VLAN. Although our analysis showed no manipulation of the shutdown sequence, we deliberately cut power to the compromised nodes as a precaution. We initiated this without prior notification to avoid unnecessary risks, but updated this status page within a few seconds once the power was cut. Locking down the VLAN and isolating the hosts successfully neutralized the threat, but temporarily severed communication with the Avoro customer area. ### Service Recovery Our Ceph storage cluster allowed us to quickly restart all affected virtual machines on unaffected replacement hardware within 5 to 30 minutes. To restore the customer area functionalities, we assigned a required communication IP address to a different host. We then systematically re-established access strictly limited to our internal dependency systems \(Avoro customer area, ISO storage, template storage, VNC, backup systems\) to ensure no vulnerabilities were exposed during the recovery process. ### Root Cause & Legacy Architecture The exploitation of the zero-day vulnerability was only possible due to two legacy conditions in the Avoro cluster: * **Public IP Accessibility:** As mentioned, two specific nodes were publicly accessible to communicate with the customer area. Our modern infrastructure is strictly isolated behind VPNs and firewalls. * **Update Dependencies:** Complex dependencies involving historically grown third-party modules in the customer area prevented us from applying host updates instantly. \(Note: The core customer area software itself is always up to date with security patches\). ### Infrastructure Rebuild & Security Enhancements **Wednesday:** Management and our technical teams outlined a permanent architectural overhaul for the Avoro customer area and the Avoro cluster. **Thursday:** We officially commenced the comprehensive rebuild. We are deploying a brand-new Proxmox cluster strictly behind our modern VPN standards and setting up a new staging instance to replace legacy third-party modules, in order to ultimately support the latest Proxmox version. To free up resources for the upcoming migrations, all Avoro VM products have temporarily been set to "out of stock". **Friday:** We started working on this postmortem. ### Customer Impact & Upcoming Maintenances Since our external network traffic analysis ruled out any data exfiltration, **you do not need to change your passwords, SSH keys, or take any other action regarding your server data.** To finalize the remediation, we will announce the following maintenances via Statuspage: * **Mid September:** Comprehensive maintenance to deploy the new Avoro customer area environment. * **Late September:** VM migrations to the newly built Avoro cluster will start. We will perform live migrations wherever technically possible, but cannot guarantee zero-downtime migrations for all individual VMs. We stand for an open error culture and acknowledge that the legacy setup of the affected hosts did not meet our current strict security standards. Thank you for your continued trust. If you have any further technical questions, our management is available via the ticket system.

Since our Edge Router Replacement at EQX-FR4, we have received a small number of reports about increased latency or timeouts when reaching certain providers. Specifically: • Vodafone: We identified congestion between one of our transit providers (Tata Communications) and Vodafone during peak traffic hours. Following our maintenance, traffic was automatically routed over this path instead of the previously used one. While the maintenance triggered this routing change, the underlying congestion was outside our network and unrelated to the maintenance itself. After the first reports, we adjusted our routing yesterday to avoid the affected path. Traffic to Vodafone is now using an unaffected alternative route. Our Smokeping reflects both the issue and the subsequent improvement. • netcup: We do not see any latency issues in our Smokeping. However, we have received at least two reports today about timeouts when accessing services hosted with netcup. We are currently analyzing the issue. Our request: If you notice anything unusual, such as latency spikes or timeouts, please reach out to our support team so we can investigate individual issues. If possible, please include an MTR in both directions (dataforest -> external network and external network -> dataforest), so that we can compare reports and identify common patterns.

If you are experiencing an issue that appears network-related but the MTR does not show any latency or packet loss, please still send us the MTR together with a reliable way to reproduce the actual problem. We need both to compare the affected path and to verify the issue itself. This can be any reproducible test, for example an SSH connection that times out or guest access to an affected application or monitoring system.

We have received and analyzed customer feedback and applied a configuration change within our backbone that may improve the stability of (very few) TCP connections previously affected by timeouts or resets. We are monitoring the situation and continue to welcome feedback on whether the change has improved the affected connections.

We have received reports of occasional packet loss affecting some TeamSpeak 3 servers over the past few minutes. We have already identified the root cause, which by design only affects traffic entering our network via our DRT-FRA8 PoP (~ 50% of the total traffic). We are working on a fix.

Our fix has proven effective. TeamSpeak 3 traffic has been stable since 23:20.

Between 1:00 and 2:00, we deployed an improved version of the previous fix for the TCP connection issue, providing better traffic distribution across our DDoS filters. The situation remains stable. As some further work is still required, we are keeping this incident open. During this work, the previous issue may briefly reoccur for a few minutes, affecting only a very small share of mostly outgoing connections (estimated <1%). The remaining work will be carried out tomorrow outside peak traffic hours.

This incident has been resolved.