dataforest Outage History

Started Jan 02, 2026, 01:46 AM UTC· resolved Jan 02, 2026, 02:11 AM UTC

1. investigating Jan 02, 2026, 02:01 AM UTC

   We are currently experiencing an outage with the vServer node epyc7513-4. Our team is actively working to resolve this issue. You can check in the control panel if you have VMs on the affected host.

2. monitoring Jan 02, 2026, 02:02 AM UTC

   The affected host is currently in recovery and all VMs are starting. We will continue to monitor the situation.

3. resolved Jan 02, 2026, 02:11 AM UTC

   All VMs are booted since 10 minutes and the host system seems to be stable again.

Read the full incident report →

Started Dec 30, 2025, 01:30 PM UTC· resolved Dec 30, 2025, 04:38 PM UTC

1. investigating Dec 30, 2025, 01:30 PM UTC

   We are receiving reports that customers with (explicitly on request) enabled symmetric DDoS protection are experiencing performance issues and lags with some applications. Regular (ingress-only) protection, active for 99% of customers, is not affected.

2. monitoring Dec 30, 2025, 02:35 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 30, 2025, 04:38 PM UTC

   The fix worked as expected, and no further problems have occurred since then. The issue was caused by the switchover of traffic to our new Equinix PoP at the end of last night's maintenance work and only affected our firstcolo location, where some of our dedicated servers and colocation racks are operated. Other products such as IP transit, VPS, and web hosting are operated at other locations and cannot have been affected by design. Customers who had explicitly requested symmetric filtering at firstcolo through our support team were affected with an approximate probability of 50%. If your dedicated server does not have "fc-fra4" in its name, it's not operated at the affected datacenter.

Read the full incident report →

Started Dec 18, 2025, 07:56 PM UTC· resolved Dec 18, 2025, 08:13 PM UTC

1. investigating Dec 18, 2025, 07:56 PM UTC

   Our panels are currently unavailable. This affects the php-friends customer panel as well as our DDoS Manager. We are investigating the problem and will provide further updates.

2. investigating Dec 18, 2025, 08:12 PM UTC

   We are continuing to investigate this issue.

3. resolved Dec 18, 2025, 08:13 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 18, 2025, 01:20 PM UTC· resolved Oct 18, 2025, 02:24 PM UTC

1. investigating Oct 18, 2025, 01:30 PM UTC

   One of the power feeds in X11 (containing only dedicated servers) has failed, while the other remains operational. Servers equipped with redundant power supplies are unaffected. Systems with only a single PSU may be impacted, depending on which feed they are connected to, resulting in roughly a 50% chance of downtime. If your server is currently unreachable, please rest assured that we are already addressing the issue. There is no need to contact our support team.

2. monitoring Oct 18, 2025, 02:06 PM UTC

   Power and servers are back online since 16:00 CEST. Should you still see any issues with your server, please check it via IPMI before reaching out to our support.

3. resolved Oct 18, 2025, 02:24 PM UTC

   The PDU of the affected feed (manufacturer: APC) was defective and has been replaced with a Panduit PDU. In this dataforest rack row, several PDUs of the same APC model have failed over the past months. Once it became clear that this was not a coincidence, we began replacing them with PDUs from another vendor that we also use in other racks within the same data center. These units have been running without issues ever since. To prevent further unexpected outages, we may proactively replace all remaining APC PDUs soon. Any such maintenance will be announced in advance.

Read the full incident report →

Started Oct 16, 2025, 11:52 PM UTC· resolved Oct 17, 2025, 11:30 PM UTC

1. identified Oct 17, 2025, 12:07 AM UTC

   One of the power feeds in X10 (containing only dedicated servers) has failed, while the other remains operational. Servers equipped with redundant power supplies are unaffected. Systems with only a single PSU may be impacted, depending on which feed they are connected to, resulting in roughly a 50% chance of downtime. If your server is currently unreachable, please rest assured that we are already addressing the issue. There is no need to contact our support team.

2. identified Oct 17, 2025, 12:48 AM UTC

   We unplugged all servers from the affected PDU, reinserted the fuse, and reconnected the servers one by one to identify which system caused the fuse to trip. Since none of the servers powered on, it became clear that the PDU itself is faulty. We are currently moving the server to another feed to boot the servers. We are also aware that this happend the second time. We are already in contact with the datacenter and the PDU manufacturer to permanently resolve the issue.

3. monitoring Oct 17, 2025, 01:02 AM UTC

   Affected servers are currently booting with the first ones already being up.

4. monitoring Oct 17, 2025, 01:18 AM UTC

   All servers are back on power. Should you still see any issues with your server, please check it via IPMI before reaching out to our support. The technicans will replace the defective PDU during the day and we need to move the servers back to the correct feed. We will keep you updated.

5. monitoring Oct 17, 2025, 05:14 PM UTC

   The PDU has been replaced and we will shut down the affected servers at 18.10.2025 - 01:00 CET, reconnect them to the correct feed, and then restart the servers.

6. resolved Oct 17, 2025, 11:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 14, 2025, 11:15 PM UTC· resolved Jan 30, 2026, 10:22 PM UTC

1. monitoring Oct 14, 2025, 11:15 PM UTC

   German version below For several days, parts of our network have experienced occasional packet loss and increased latency, usually lasting a few seconds, sometimes up to a minute. Some days it happens two or three times. Other days it does not happen at all. The cause is ongoing DDoS attacks driven by the “Aisuru” botnet, which is pushing many networks worldwide to their capacity limits. Alongside us, many other large cloud and hosting providers are affected, including some with significantly larger networks. Trade press reports we link below estimate the botnet’s capacity at 20-30 Tbps. Because the attacks target completely different customers and regularly our entire network (“carpet bombing”) at record bandwidths, effective filtering is very complex. Your data is not at risk: the attacks aim solely to overload servers and network infrastructure. Since the first attacks, we have been working almost 24/7 on various countermeasures. We have improved our filtering and detection mechanisms, expanded capacity, and used traffic engineering to distribute attacks across our upstreams to prevent overload on individual uplinks. We are also working closely with our transit providers on pre-filtering and early detection. These measures are now having an effect – the vast majority of attacks are fully filtered and no longer cause damage. However, it is unavoidable that new measures must occasionally be deployed. During that time, latency-sensitive applications like game servers may experience issues. For TCP-based applications like web and mail servers, outages are usually not observed because these applications are not sensitive to short packet loss or latency spikes. We would like to communicate transparently that it will take several months to fully implement all measures and network expansions. Unfortunately, upgrades in the terabit range, as well as the provisioning of cross-connects and dark fibers, involve lead times that we cannot avoid. We use these unavoidable “waiting periods” to implement other measures, prepare the planned upgrades, and bring new network hardware into operation ahead of schedule, so that you will generally not notice any issues. The situation improves on a daily basis, and most days and weeks will pass completely without disruption. Rest assured: we are working continuously to restore the usual dataforest quality, and we thank you for your understanding, the motivating words, and your loyalty. A detailed blog post about our network expansion was already planned and will now appear a bit earlier. Over the coming months, various network maintenance activities will be required, some of which may be scheduled with short notice. We will keep you informed via our status page. This status post will also remain open even if we do not expect or observe any (negative) impact over longer periods of time. Please make sure to contact us as early as possible if you notice any unexpected issues lasting longer than a few seconds. German Seit einigen Tagen kommt es in Teilen unseres Netzwerks zu gelegentlichen Paketverlusten und erhöhten Latenzen, die wenige Sekunden bis maximal eine Minute andauern. An manchen Tagen passiert dies zwei- oder dreimal, an anderen gar nicht. Auslöser sind anhaltende DDoS-Angriffe, verursacht durch das “Aisuru”-Botnetz, das viele Netzwerke weltweit an ihre Kapazitätsgrenzen bringt. Neben uns sind viele andere große Cloud- und Hostinganbieter mit teilweise noch erheblich größeren Netzwerken betroffen. In den Fachmedien finden sich hierzu bereits verschiedene Berichte, in denen die Kapazität des Botnetzes mit 20-30 Tbps beziffert wird, die wir unten verlinkt haben. Da sich die Angriffe gegen völlig unterschiedliche Kunden und regelmäßig unser gesamtes Netzwerk (“Carpet Bombing”) richten und mit Rekordbandbreiten ausgeführt werden, ist eine effektive Filterung sehr komplex. Ihre Daten sind nicht in Gefahr: Die Angriffe zielen rein auf die Überlastung von Servern und Netzwerkinfrastruktur ab. Wir arbeiten seit den ersten Angriffen fast 24/7 an verschiedensten Maßnahmen zu deren Abwehr. So wurden unsere Filter- und Detektionsmechanismen verbessert, Kapazitäten erweitert und durch Traffic-Engineering sichergestellt, dass sich die Angriffe möglichst ideal auf unsere Upstreams verteilen, damit es zu keinen Überlastungen einzelner Uplinks mehr kommt. Auch arbeiten wir mit unseren Transitprovidern intensiv an der Vorfilterung und Früherkennung dieser Angriffe. Diese Maßnahmen zeigen inzwischen Wirkung – die allermeisten Angriffe werden vollständig gefiltert und richten keinen Schaden mehr an. Allerdings ist unvermeidbar, dass gelegentlich neue Maßnahmen ergriffen werden müssen. Währenddessen kann es insbesondere bei latenzkritischen Anwendungen wie Gameservern zu Problemen kommen. Bei TCP-basierten Anwendungen wie Web- und Mailservern kann in der Regel kein Ausfall registriert werden, weil diese Anwendungen nicht empfindlich auf kurze Paketverluste oder Latenzspitzen reagieren. Wir möchten transparent kommunizieren, dass es mehrere Monate dauern wird, bis alle Maßnahmen und Netzwerkerweiterungen umgesetzt sind. Leider sind Upgrades im Terabit-Bereich, Provisionierungen von Crossconnects und Glasfaserstrecken mit Vorlaufzeiten verbunden, die wir nicht umgehen können. Entsprechende “Wartezeiten” nutzen wir zur Realisierung anderer Maßnahmen, Vorbereitung der geplanten Upgrades sowie die vorgezogene Inbetriebnahme neuer Netzwerkhardware, sodass Sie in aller Regel keine Probleme feststellen können werden, die Situation täglich besser wird und die meisten Tage und Wochen völlig störungsfrei verlaufen. Seien Sie versichert: Wir arbeiten permanent daran, die gewohnte dataforest-Qualität wiederherzustellen, und bedanken uns für Ihr Verständnis, die motivierenden Worte und Ihre Treue. Ein ausführlicher Blogartikel zum Ausbau unseres Netzwerks war ohnehin geplant und wird nun etwas früher erscheinen. In den nächsten Monaten werden verschiedene Wartungsarbeiten am Netzwerk mit teils kurzer Vorlaufzeit notwendig sein. Wir halten Sie auf unserer Statusseite auf dem Laufenden. Ebenso halten wir diesen Statuspost bis auf Weiteres offen, auch wenn wir über längere Zeiträume keine (negativen) Auswirkungen erwarten oder feststellen. Bitte melden Sie sich unbedingt frühzeitig bei uns, wenn Sie unerwartete Probleme feststellen, die über wenige Sekunden hinausgehen. Media reports / Medienberichte: Aisuru’s 30 Tbps botnet traffic crashes through major US ISPs Another record-breaking DDoS? Aisuru botnet suspected behind 29.69 Tbps gaming outages US ISP-hosted IoT devices fuel Aisuru DDoS botnet DDoS Botnet Aisuru Blankets US ISPs in Record DDoS Researchers Warn RondoDox Botnet is Weaponizing Over 50 Flaws Across 30+ Vendors

2. monitoring Oct 23, 2025, 11:35 AM UTC

   We are seeing a major outage since one minute.

3. monitoring Oct 23, 2025, 12:03 PM UTC

   Connectivity was restored at 14:00 CEST, BGP sessions are coming up since then. A software bug in our aggregation switches led to a major outage when we committed a (regular) change in our edge network related to the DDoS attacks. Only rebooting the devices could restore Layer 2 connectivity which was completely lost. A post-mortem will follow.

4. monitoring Nov 02, 2025, 09:56 AM UTC

   The situation has remained stable for over a week with no disruptions or packet loss, despite ongoing daily attacks. This incident will remain open until the post-mortem for last week’s outage has been published.

5. monitoring Nov 02, 2025, 01:35 PM UTC

   Between 13:26 and 13:27 (CET) packet loss occurred in our network due to a changed attack vector. The situation has been stable since then, and additional measures have already been implemented in the meantime.

6. monitoring Dec 12, 2025, 12:33 AM UTC

   Two days ago, we published a post-mortem on our blog for the October 23, 2025 network outage. The article also discusses the network upgrades announced here on our status page, in particular the addition of another PoP (Point of Presence). German version: dataforest.net/blog/post-mortem-23-10-25 English version: dataforest.net/en/blog/post-mortem-23-10-25

7. monitoring Dec 27, 2025, 04:25 AM UTC

   Yesterday, we received customer reports indicating that HTTPS connections to and from the United States – particularly involving destinations geographically close to Chicago – experienced intermittent issues for a few hours. Based on our analysis, the scope was very limited: Most customers reported alerts from external monitoring probes located in the US, while their real visitors and production traffic were unaffected. In a smaller number of cases, specific endpoints (e.g., Docker registries hosted in the US) could not be reached reliably, again without broader customer-facing impact. Only IPv4 traffic was affected. As soon as the first reports came in, we began an immediate analysis and traced the behavior to the activation of additional network protection measures. To minimize further impact, we reverted these changes and continued debugging during reactivation. We implemented a corrective change at the start of tonight’s maintenance window announced under https://status.dataforest.net/incidents/tbtvg8bg6ct6, which resolved the issue permanently.

8. monitoring Dec 27, 2025, 11:09 PM UTC

   193.46.81.0/24 is currently unreachable, we are working on it.

9. monitoring Dec 27, 2025, 11:32 PM UTC

   193.46.81.0/24 is reachable again.

10. resolved Jan 30, 2026, 10:22 PM UTC

    DE Die Situation ist seit vielen Wochen stabil. Die im Blog beschriebenen Netzwerkumbauten und -Upgrades sind größtenteils umgesetzt, wie Sie möglicherweise bereits den letzten Wartungsarbeiten entnommen haben. Die restlichen Arbeiten schließen wir voraussichtlich (wie geplant) bis Ende Februar ab. Um die Statusseite übersichtlich zu halten, schließen wir diesen Eintrag hiermit. Wir erwarten keine weiteren relevanten Probleme. EN The situation has been stable for many weeks. The network modifications and upgrades described in the blog have largely been implemented, as you may have already noticed from the latest maintenance work. We expect to complete the remaining work (as planned) by the end of February. To keep the status page clear, we are closing this entry. We do not expect any further relevant issues.

Read the full incident report →

Started Oct 10, 2025, 02:37 PM UTC· resolved Oct 11, 2025, 08:08 PM UTC

1. identified Oct 10, 2025, 02:37 PM UTC

   We observed packet loss between 16:30 and 16:33 CEST. The situation is stable since then.

2. identified Oct 10, 2025, 05:35 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved Oct 11, 2025, 08:08 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 27, 2025, 02:21 PM UTC· resolved Aug 27, 2025, 07:39 PM UTC

1. monitoring Aug 27, 2025, 02:21 PM UTC

   Starting at 16:12 CEST, we received large DDoS attacks via our DTAG (Deutsche Telekom AG) port, leading to packet loss for DTAG customers. At 16:18, we have withdrawn our IPv4 announcements to DTAG, which solved the issue immediately.

2. resolved Aug 27, 2025, 07:39 PM UTC

   Since our initial update, no further packet loss has affected DTAG (Deutsche Telekom) customers. Traffic telemetry confirms brief congestion on the DTAG port from 16:11–16:14 and 16:17–16:18 CEST. We have re-exported our IPv4 prefixes to DTAG a few hours ago so traffic takes the direct path again. To avoid port congestion, we implemented additional traffic engineering policies. Until today, rules such as not exporting to DTAG peers and AS path prepending for DTAG transit customers were not applied to all prefixes. They are now applied consistently across all prefixes, including those of our BGP customers / downstreams, ensuring maximum stability of our DTAG port.

Read the full incident report →

Started Aug 24, 2025, 07:15 AM UTC· resolved Aug 24, 2025, 11:08 AM UTC

1. identified Aug 24, 2025, 09:36 AM UTC

   The issue has been identified and a fix is being implemented.

2. identified Aug 24, 2025, 09:53 AM UTC

   Our PHP-Friends billing system is currently unreachable. As a result, crm.php-friends.de is unreachable, too, which makes it impossible to login to ccp.dataforest.net (while existing sessions are unaffected). Placing new orders at php-friends.de is also not possible till the incident has been resolved.

3. resolved Aug 24, 2025, 11:08 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 09, 2025, 09:27 PM UTC· resolved Aug 11, 2025, 06:08 PM UTC

1. identified Aug 09, 2025, 09:31 PM UTC

   We observed unusual CPU load on several DDoS filters, resulting in packet loss. The filters were taken out of service within a minute, and we are investigating the root cause. The issue has since been resolved and there is currently no ongoing impact. Only IPv4 traffic with active DDoS protection was affected.

2. monitoring Aug 09, 2025, 10:08 PM UTC

   We have deployed a software fix to our DDoS protection cluster and brought the filters back online. No further issues could be observed since our initial status post. We are monitoring the situation.

3. resolved Aug 11, 2025, 06:08 PM UTC

   Since implementation of the fix, the situation is stable, no further packet loss occured, so we are closing the incident. Some minor issues with the new software version have been fixed yesterday.

Read the full incident report →

Started Aug 05, 2025, 04:37 PM UTC· resolved Aug 05, 2025, 06:46 PM UTC

1. identified Aug 05, 2025, 04:50 PM UTC

   One of the power feeds in rack X13 has failed, while the other remains operational. Servers equipped with redundant power supplies are unaffected. Systems with only a single PSU may be impacted, depending on which feed they are connected to, resulting in roughly a 50% chance of downtime. If your server is currently unreachable, please rest assured that we are already addressing the issue. There is no need to contact our support team.

2. identified Aug 05, 2025, 06:04 PM UTC

   We unplugged all servers from the affected PDU, reinserted the fuse, and reconnected the servers one by one to identify which system caused the fuse to trip. Since none of the servers powered on, it became clear that the PDU itself is faulty. It will be replaced as soon as possible.

3. resolved Aug 05, 2025, 06:46 PM UTC

   All servers came back online between 20:30 and 20:45 CEST according to our monitoring. Unfortunately, the data center was not able to replace the defective PDU immediately. As a temporary measure, we connected the affected servers to a nearby PDU in another rack. These systems will need to be reconnected to the correct PDU once it has been replaced, which will result in approximately five minutes of downtime. We will notify all affected customers in advance.

Read the full incident report →

Started Aug 05, 2025, 10:59 AM UTC· resolved Aug 05, 2025, 01:58 PM UTC

1. investigating Aug 05, 2025, 10:59 AM UTC

   Our monitoring systems detected short periods of packet loss to some destinations within our maincubes FRA01 location at 11:47:42, 12:15:34, 12:18:02, and 12:24:23 CEST, each lasting around 15 seconds. While the situation has been stable since then, we're still looking into the root cause. Initial findings point to log messages on one of the two redundant core switches involved, while the other didn't log anything unusual. This suggests that the impact for customers was likely minimal, but we're continuing our investigation and will keep the incident open until it's fully clarified and mitigated.

2. identified Aug 05, 2025, 12:24 PM UTC

   We have identified the root cause and will carry out an emergency configuration change in the next 60 minutes. This may result in a brief period of packet loss but is necessary to ensure stable operation.

3. resolved Aug 05, 2025, 01:58 PM UTC

   The configuration change was successfully applied with minimal impact at 15:18 / 15:27 CEST. The root cause was an inconsistent Spanning Tree configuration within our maincubes FRA01 core network, which has now been resolved. Only a limited number of racks were affected, depending on the specific configuration of their Top-of-Rack switches. Over the coming weeks and months, we will review the configuration of all devices involved in the topology to identify potential improvements. Should further changes be necessary, we will implement them during announced maintenance windows at night, as the immediate issue has been resolved and no further disruptions are expected.

Read the full incident report →

Started Aug 02, 2025, 07:45 AM UTC· resolved Aug 02, 2025, 11:27 AM UTC

1. investigating Aug 02, 2025, 07:45 AM UTC

   Since 10 minutes we begin to see IPv6 issues in one VLAN, mainly containing PHP-Friends SSD G3 VMs including managed servers, some Plesk webhosting VMs and a few Avoro VMs. Details will follow shortly. If your IPv6 connection is working, you are unaffected.

2. monitoring Aug 02, 2025, 08:02 AM UTC

   Situation is stable since 5-10 minutes depending on the specific IP, we will follow-up shortly about the details of the incident.

3. resolved Aug 02, 2025, 11:27 AM UTC

   At 09:34 CEST, our on-call engineer was informed about reachability issues affecting some VMs in the following IPv6 prefixes: 2a0d:5940:3::/48 2a0d:5940:6::/48 2a0d:5940:4::/48 2a0d:5941:1::/48 These prefixes contain G3 VMs (PHP-Friends), including managed servers, our Plesk web servers, and only a few Avoro VMs. As no cause could be found, the incident was additionally escalated to our network team at 09:44 and posted to our status page. At 09:52, the IPv6 checks began to recover. The following learnings emerged: - Not all VMs in the mentioned prefixes were affected by the issue, as it required a specific configuration to be vulnerable. If you did not notice any issues with your IPv6 connectivity, you were not affected. - Escalation should have happened faster. - Our IPv6 setup for virtual servers requires additional work to avoid such incidents entirely (best case) or at least to significantly improve the time to repair. We will work on this over the next two weeks. Any maintenance required for this (most likely affecting our web servers) will be announced on our status page and carried out as carefully and with as little impact as possible.

Read the full incident report →

Started Jul 24, 2025, 02:45 PM UTC· resolved Jul 24, 2025, 06:01 PM UTC

1. identified Jul 24, 2025, 02:45 PM UTC

   We need to perform urgent maintenance on our Anti-DDoS infrastructure to address an identified issue that requires immediate action to prevent potential service impact. During this period, only the basic pre-filtering of our routing infrastructure will be active, while the granular DDoS filters will be temporarily unavailable. No service disruption is expected.

2. monitoring Jul 24, 2025, 03:24 PM UTC

   When we turned the DDoS filters back into service 5 minutes ago, timeouts of some existing TCP and UDP sessions occured due to human error. The root cause is already identified and resolved.

3. monitoring Jul 24, 2025, 03:35 PM UTC

   Further investigation of the issue mentioned in our previous update showed that only IPv4 addresses with an active DDoS mitigation profile were affected, and timeouts occurred for just a few seconds.

4. resolved Jul 24, 2025, 06:01 PM UTC

   The maintenance was successfully completed with the final update posted at 17:35 CEST.

Read the full incident report →

Started Jul 24, 2025, 04:24 AM UTC· resolved Jul 24, 2025, 04:30 AM UTC

1. investigating Jul 24, 2025, 04:24 AM UTC

   There is currently a partial inaccessibility of individual IP addresses in the networks 2.59.132.0/24 and 2.59.133.0/24. The problem exists since 06:09 and already occurred for some minutes at 04:30. We are investigating.

2. resolved Jul 24, 2025, 04:30 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 23, 2025, 01:27 PM UTC· resolved Jul 23, 2025, 05:02 PM UTC

1. investigating Jul 23, 2025, 01:27 PM UTC

   We are currently investigating intermittent port flaps between our Edge routing (Interxion FRA8) and our location firstcolo FRA4. These events may temporarily reduce redundancy and cause increased latency. At this time, we have not observed any packet loss, but we are working to stabilize the situation promptly to ensure continued reliability, especially during peak hours. While we are troubleshooting, there is an increased risk of service impact at this datacenter. We will provide an update as soon as more details are available.

2. identified Jul 23, 2025, 04:05 PM UTC

   The situation remains stable, with no packet loss observed, but we continue to see increases in latency. We currently suspect a faulty transceiver as the root cause. Since multiple components are involved (routers, switches, converters, DWDM equipment) across two different datacenters, identifying and replacing the defective part requires a step-by-step process and therefore takes time. Our teams are working in parallel at both datacenters to resolve this as quickly as possible. To further improve redundancy and capacity, we already ordered additional dark fibers and opened a new PoP at Equinix FR4 a few weeks ago in response to bandwidth growth. These links have a lead time of approximately 8–10 weeks. Once they are in service, a failure on a single line will no longer cause any impact at all, as intended.

3. resolved Jul 23, 2025, 05:02 PM UTC

   After migrating the transport service to alternative interfaces on the muxponders in both datacenters, the situation has stabilized. Latencies have decreased and are no longer spiking. Since the devices did not report any errors, we had to replace nearly all optical transceivers involved before taking this final (and resolving) action. The issue has now been resolved for about 15 minutes without any recurrence.

Read the full incident report →

Started Jul 14, 2025, 10:00 PM UTC· resolved Jul 14, 2025, 11:14 PM UTC

1. monitoring Jul 14, 2025, 10:20 PM UTC

   We are observing an outage of Cloudflare's public DNS resolver (1.1.1.1). Also we see a traffic drop on the Private Peering (PNI) we have with Cloudflare. Therefore, customers using Cloudflare DNS might face issues. While this is not an issue within the dataforest infrastructure, we are posting this announcement as wide parts of the Internet rely on Cloudflare DNS services, which is why the impact of such an outage is potentially heavy. If your server is using Cloudflare as a DNS resolver, you might switch to another DNS service like Google's 8.8.8.8 which is default for our dedicated and virtual servers since it has been extremely reliable.

2. resolved Jul 14, 2025, 11:14 PM UTC

   According to https://www.cloudflarestatus.com/incidents/28r0vbbxsh8f, the issue has been fixed. We can confirm that.

Read the full incident report →

Started Jun 14, 2025, 11:12 AM UTC· resolved Jul 30, 2025, 02:29 PM UTC

1. monitoring Jun 14, 2025, 11:12 AM UTC

   We are currently aware of an issue affecting some customers running Windows Server 2025. After recent Windows Updates, systems with certain AMD processors or running in virtualized environments may experience startup failures or blue screen errors with the message "UNSUPPORTED PROCESSOR." This issue is caused by the latest Windows Update and is not related to our services or infrastructure. We recommend affected customers skip or defer the problematic update if possible. If you are experiencing these symptoms, please contact our support team for further assistance. We are closely monitoring the situation and will provide updates as more information becomes available. Thank you for your understanding.

2. resolved Jul 30, 2025, 02:29 PM UTC

   Microsoft has released an update that resolves the issue.

Read the full incident report →

Started Jun 13, 2025, 10:42 AM UTC· resolved Jun 13, 2025, 11:22 AM UTC

1. resolved Jun 13, 2025, 11:55 AM UTC

   At 12:42 CEST, a faulty configuration change on our maincubes FRA01 router was committed, leading to an unavailability of the default route for BGP customers. The issue was detected and fixed at 13:22. Customers receiving a fulltable were unaffected as well as customers not speaking BGP with us at all which is the case for almost every customer.

Read the full incident report →

Started Jun 08, 2025, 04:07 PM UTC· resolved Jun 08, 2025, 04:14 PM UTC

1. identified Jun 08, 2025, 04:12 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved Jun 08, 2025, 04:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 03, 2025, 09:17 AM UTC· resolved Jun 03, 2025, 10:06 AM UTC

1. investigating Jun 03, 2025, 09:17 AM UTC

   We are currently investigating this issue.

2. identified Jun 03, 2025, 09:23 AM UTC

   We just performed a emergency reboot and keep you updated.

3. monitoring Jun 03, 2025, 09:25 AM UTC

   The system is booted up, all VM guests are booting now too, we keep monitor the system status

4. resolved Jun 03, 2025, 10:06 AM UTC

   We keep monitor the situation but for now mark it as resolved.

Read the full incident report →

Started May 20, 2025, 06:10 AM UTC· resolved May 20, 2025, 06:22 AM UTC

1. investigating May 20, 2025, 06:10 AM UTC

   There is currently a partial power failure in one of our racks. If your dedicated server has been offline since approx. 07:30 CET and cannot be managed in the web interface, then it is affected by the outage.

2. identified May 20, 2025, 06:20 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved May 20, 2025, 06:22 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 22, 2025, 12:57 AM UTC· resolved Apr 22, 2025, 03:19 AM UTC

1. investigating Apr 22, 2025, 01:05 AM UTC

   We are currently experiencing an outage of the EPYC Rootserver node epyc9754-2. Our team is actively working to resolve this issue. You can check in the control panel if you have VMs on the affected host.

2. monitoring Apr 22, 2025, 02:58 AM UTC

   Unfortunately, all recovery attempts for the affected node were unsuccessful – the hardware is faulty and will be repaired later. Thanks to our redundant Ceph storage, affected VMs can be started on other nodes. We’ve already begun the migration and the first VMs are back online.

3. resolved Apr 22, 2025, 03:19 AM UTC

   All VMs were started on replacement hardware.

Read the full incident report →