dataforest Outage History

Started Oct 22, 2026, 11:15 AM UTC· resolved Oct 22, 2026, 05:15 PM UTC

1. monitoring Oct 22, 2026, 11:15 AM UTC

   Over the past few hours, we have observed increased disk latency in our Ceph cluster, affecting storage performance in both the dataforest cloud and the Avoro cluster. We have identified a faulty disk and taken it out of service. If you continue to observe unusual disk latency or performance issues, please report them to [email protected]. We are not aware of any resulting disruptions to production services or outages. However, customers may have noticed the increased latency in their own monitoring, so we are sharing this update to explain what happened. We will also improve our monitoring to detect similar issues sooner.

2. resolved Oct 22, 2026, 05:15 PM UTC

   No more issues were observed. Therefore we close this incident.

Read the full incident report →

Started Sep 26, 2026, 05:07 PM UTC· resolved Sep 26, 2026, 11:50 PM UTC

1. identified Sep 26, 2026, 05:07 PM UTC

   The subnet 104.167.24.0/24 is unreachable since 18:57 due to a carpet bomb DDoS attack. We are working on the issue.

2. identified Sep 26, 2026, 05:12 PM UTC

   45.145.40.0/24 is unreachable since 19:10, too. Connectivity for 104.167.24.0/24 has been restored at 19:09.

3. identified Sep 26, 2026, 05:15 PM UTC

   45.145.40.0/24 is back online since 19:13, too. We have applied first measures to reduce the impact of the still running attacks and will apply additional traffic engineering later to ensure an equal traffic distribution across our POPs. Our backbone capacity is sufficient to mitigate these attacks, but traffic distribution needs to be optimized.

4. resolved Sep 26, 2026, 11:50 PM UTC

   Reason for outage: After the recent edge router replacement at DRT-FRA8, some attack traffic was not distributed equally across both POPs for various reasons, and blackholing / prefix withdrawal was triggered for the mentioned prefixes. Our fixes applied shortly afterward have proven effective, and no further disruptions occurred despite ongoing attacks.

Read the full incident report →

Started Sep 23, 2026, 06:36 PM UTC· resolved Sep 25, 2026, 03:52 AM UTC

1. investigating Sep 23, 2026, 06:36 PM UTC

   Between 20:18 and 20:23, we observed two brief periods of packet loss at one of our two PoPs, which affected some connections. Everything has been stable since then, and we're investigating the cause.

2. monitoring Sep 23, 2026, 10:30 PM UTC

   An issue at our EQX-FR4 PoP caused brief packet loss between 20:17:30 and 20:22:30 CEST. Traffic handled through our DRT-FRA8 PoP continued to operate normally throughout the incident, meaning a large part of our overall traffic was not affected. Across our network, total traffic dropped by around 40% for approximately one minute at the beginning of the incident. During recovery, we observed another much shorter drop of around 40% between 20:20:26 and 20:20:38. Within the following two minutes, traffic returned to normal levels. The issue was alerted internally at 20:22, followed by an incident call shortly after. During the investigation, we identified a likely root cause and implemented a first fix. The situation has remained stable since then.

3. resolved Sep 25, 2026, 03:52 AM UTC

   We have validated our findings, and the situation has remained fully stable.

Read the full incident report →

Started Sep 14, 2026, 02:47 PM UTC· resolved Sep 14, 2026, 08:31 PM UTC

1. monitoring Sep 14, 2026, 02:47 PM UTC

   At 16:25, 16:32, and 16:35, a particular DDoS attack overloaded the DDoS filters at both POPs, causing temporary packet loss. A temporary fix has been implemented, and we are working on a permanent solution. Unprotected traffic was unaffected.

2. monitoring Sep 14, 2026, 03:05 PM UTC

   We have updated the initial version of this post, as we originally assumed that only one of our POPs was affected. In fact, both were.

3. monitoring Sep 14, 2026, 06:50 PM UTC

   The issue recurred at 20:39 for a few seconds, but with less impact as only one of our POPs was affected. We are still working on a permanent solution; our temporary fix has already reduced the impact.

4. resolved Sep 14, 2026, 08:31 PM UTC

   At the previously specified times, an incorrect handling of dynamic attack pre-filtering during specific, very large DDoS attacks overwhelmed some of our filtering appliances, causing the issues described in this post. A permanent fix was deployed at 21:00, which has since been successfully verified under real-world traffic conditions. We can confirm that no further issues occurred other than those detailed here.

Read the full incident report →

Started Sep 01, 2026, 11:01 AM UTC· resolved Sep 01, 2026, 11:10 PM UTC

1. identified Sep 01, 2026, 11:01 AM UTC

   Two of our Avoro VM hosts, epyc7513-1 and epyc9374f-8, are currently experiencing an outage. The majority of Avoro VMs are not affected by the host outage itself. Customers can check the customer area to see whether any of their VMs are running on the affected hosts. VM control functions such as starting, stopping, and reinstalling are currently unavailable for all Avoro customers, as this functionality depends on the affected hosts. Affected VMs will be restarted on replacement hardware as soon as possible. We expect this process to take up to two hours. Individual VMs may be restored significantly earlier, as they will be brought back online sequentially. We will share further details as soon as they become available.

2. monitoring Sep 01, 2026, 11:38 AM UTC

   All affected VMs have been back online on replacement hardware for the past 10 minutes. Restoration was completed in well under 30 minutes, significantly faster than the previously estimated recovery time of up to two hours. We are now working to restore the control panel functionality.

3. monitoring Sep 01, 2026, 05:44 PM UTC

   Basic control panel functionality was restored at 18:50, including starting and stopping VMs as well as booting from mounted ISOs. Creating and restoring backups is also possible again. We are continuing to work on restoring VNC access and the reinstallation feature.

4. resolved Sep 01, 2026, 11:10 PM UTC

   VNC access was restored at 22:37, followed by the reinstallation feature at 01:00. All previously affected control panel functionality has now been restored. Thank you for bearing with us while we worked to restore these services.

5. postmortem Sep 06, 2026, 09:52 PM UTC

   Following the incident on Tuesday, September 1, 2026, we want to provide a comprehensive postmortem regarding the root cause, our immediate response, and the resulting infrastructure upgrades. ### Incident Discovery & Scope On Tuesday morning, we detected unauthorized access to 2 of our approximately 50 Avoro virtualization hosts \(epyc7513-1 and epyc9374f-8\). Because this intrusion exploited a design flaw unique to this specific legacy setup, all other products \(such as PHP-Friends vServer, the dataforest cloud, and our internal systems\) operating in completely separate clusters were never at risk. By analyzing network traffic externally – completely outside the compromised systems – and monitoring active processes on the hosts, we confirmed the intrusion was strictly a known crypto-mining script. It exploited a Proxmox zero-day vulnerability disclosed just 90 minutes prior. We verified there was no ransomware, and no customer data or disk images were accessed or exfiltrated. ### Immediate Response & Control Panel Disconnect The Avoro customer area \(our control panel based on WHMCS\) historically required external communication with the cluster. For failover purposes, exactly the two affected nodes were equipped with a public IPv4 address to handle this traffic. Once we identified the intrusion, we completely locked down the Proxmox management VLAN. Although our analysis showed no manipulation of the shutdown sequence, we deliberately cut power to the compromised nodes as a precaution. We initiated this without prior notification to avoid unnecessary risks, but updated this status page within a few seconds once the power was cut. Locking down the VLAN and isolating the hosts successfully neutralized the threat, but temporarily severed communication with the Avoro customer area. ### Service Recovery Our Ceph storage cluster allowed us to quickly restart all affected virtual machines on unaffected replacement hardware within 5 to 30 minutes. To restore the customer area functionalities, we assigned a required communication IP address to a different host. We then systematically re-established access strictly limited to our internal dependency systems \(Avoro customer area, ISO storage, template storage, VNC, backup systems\) to ensure no vulnerabilities were exposed during the recovery process. ### Root Cause & Legacy Architecture The exploitation of the zero-day vulnerability was only possible due to two legacy conditions in the Avoro cluster: * **Public IP Accessibility:** As mentioned, two specific nodes were publicly accessible to communicate with the customer area. Our modern infrastructure is strictly isolated behind VPNs and firewalls. * **Update Dependencies:** Complex dependencies involving historically grown third-party modules in the customer area prevented us from applying host updates instantly. \(Note: The core customer area software itself is always up to date with security patches\). ### Infrastructure Rebuild & Security Enhancements **Wednesday:** Management and our technical teams outlined a permanent architectural overhaul for the Avoro customer area and the Avoro cluster. **Thursday:** We officially commenced the comprehensive rebuild. We are deploying a brand-new Proxmox cluster strictly behind our modern VPN standards and setting up a new staging instance to replace legacy third-party modules, in order to ultimately support the latest Proxmox version. To free up resources for the upcoming migrations, all Avoro VM products have temporarily been set to "out of stock". **Friday:** We started working on this postmortem. ### Customer Impact & Upcoming Maintenances Since our external network traffic analysis ruled out any data exfiltration, **you do not need to change your passwords, SSH keys, or take any other action regarding your server data.** To finalize the remediation, we will announce the following maintenances via Statuspage: * **Mid September:** Comprehensive maintenance to deploy the new Avoro customer area environment. * **Late September:** VM migrations to the newly built Avoro cluster will start. We will perform live migrations wherever technically possible, but cannot guarantee zero-downtime migrations for all individual VMs. We stand for an open error culture and acknowledge that the legacy setup of the affected hosts did not meet our current strict security standards. Thank you for your continued trust. If you have any further technical questions, our management is available via the ticket system.

Read the full incident report →

Started Aug 29, 2026, 11:17 AM UTC· resolved Aug 31, 2026, 09:16 PM UTC

1. investigating Aug 29, 2026, 11:17 AM UTC

   Since our Edge Router Replacement at EQX-FR4, we have received a small number of reports about increased latency or timeouts when reaching certain providers. Specifically: • Vodafone: We identified congestion between one of our transit providers (Tata Communications) and Vodafone during peak traffic hours. Following our maintenance, traffic was automatically routed over this path instead of the previously used one. While the maintenance triggered this routing change, the underlying congestion was outside our network and unrelated to the maintenance itself. After the first reports, we adjusted our routing yesterday to avoid the affected path. Traffic to Vodafone is now using an unaffected alternative route. Our Smokeping reflects both the issue and the subsequent improvement. • netcup: We do not see any latency issues in our Smokeping. However, we have received at least two reports today about timeouts when accessing services hosted with netcup. We are currently analyzing the issue. Our request: If you notice anything unusual, such as latency spikes or timeouts, please reach out to our support team so we can investigate individual issues. If possible, please include an MTR in both directions (dataforest -> external network and external network -> dataforest), so that we can compare reports and identify common patterns.

2. investigating Aug 29, 2026, 01:32 PM UTC

   If you are experiencing an issue that appears network-related but the MTR does not show any latency or packet loss, please still send us the MTR together with a reliable way to reproduce the actual problem. We need both to compare the affected path and to verify the issue itself. This can be any reproducible test, for example an SSH connection that times out or guest access to an affected application or monitoring system.

3. monitoring Aug 29, 2026, 05:00 PM UTC

   We have received and analyzed customer feedback and applied a configuration change within our backbone that may improve the stability of (very few) TCP connections previously affected by timeouts or resets. We are monitoring the situation and continue to welcome feedback on whether the change has improved the affected connections.

4. identified Aug 29, 2026, 09:16 PM UTC

   We have received reports of occasional packet loss affecting some TeamSpeak 3 servers over the past few minutes. We have already identified the root cause, which by design only affects traffic entering our network via our DRT-FRA8 PoP (~ 50% of the total traffic). We are working on a fix.

5. monitoring Aug 29, 2026, 10:45 PM UTC

   Our fix has proven effective. TeamSpeak 3 traffic has been stable since 23:20.

6. monitoring Aug 30, 2026, 12:59 PM UTC

   Between 1:00 and 2:00, we deployed an improved version of the previous fix for the TCP connection issue, providing better traffic distribution across our DDoS filters. The situation remains stable. As some further work is still required, we are keeping this incident open. During this work, the previous issue may briefly reoccur for a few minutes, affecting only a very small share of mostly outgoing connections (estimated <1%). The remaining work will be carried out tomorrow outside peak traffic hours.

7. resolved Aug 31, 2026, 09:16 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 16, 2026, 08:33 AM UTC· resolved Aug 17, 2026, 12:22 PM UTC

1. investigating Aug 16, 2026, 08:58 AM UTC

   We are currently investigating this issue.

2. monitoring Aug 16, 2026, 09:06 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 17, 2026, 12:22 PM UTC

   Yesterday morning at 10:33, our on-call engineer was alerted to increased latency at our FC-FRA4 location. Initial analysis suggested a false positive, as the issue affected specific network paths to the datacenter rather than all routes. After additional alerts for brief packet loss arose, our engineer escalated the incident to our NOC at 10:48. A partially mitigated carpet-bombing DDoS attack was identified as the root cause, and this status post was published at 10:58. Full mitigation was active by 11:02. Root Cause & Impact Traffic flowing from our EQX-FR4 PoP to FC-FRA4 experienced link congestion. Traffic via DRT-FRA8 was unaffected because the attack was properly filtered at that site. Consequently, approximately 50% of incoming routes were impacted by brief packet loss and latency spikes. Next Steps Following today's internal review, we are implementing both technical and administrative improvements: • Mitigation Bypasses: Implementing additional measures to prevent attack traffic from reaching the datacenter when real-time mitigation fails. • Escalation Process: Refining internal procedures to accelerate incident handling.

Read the full incident report →

Started Aug 15, 2026, 11:00 AM UTC· resolved Aug 15, 2026, 04:56 PM UTC

1. identified Aug 15, 2026, 11:00 AM UTC

   We want to inform our customers about an ongoing incident with one of our transit providers, GTT Communications, which is currently not causing a negative impact on our network. In accordance with our communication standards, we decided to still publish it as a precautionary measure. At 05:19, our NOC was alerted that one of our 400G ports with GTT was down at our EQX-FR4 location. During diagnosis, we found the issue to be on GTT's side and created a ticket with their NOC. Right after creation, at 05:51, a second 400G port went offline. Neither outage caused any visible disruption according to our real-time network overview, as convergence happens almost instantly if a router port goes down physically. At 06:36, the GTT NOC claimed that the issue was outside of their network and advised us to check our cross-connects with the datacenter operator (Equinix). At 06:56, the two remaining 400G ports we have with GTT at EQX-FR4 went down, causing about 15 seconds of packet loss on a few routes because the ports failed simultaneously and GTT traffic shifted automatically to our second PoP at DRT-FRA8. We escalated the matter to GTT immediately, but did not receive feedback for several hours. At 11:21, we were informed that the reason for the outage is an ongoing cooling system failure at EQX-FR5, where two water leaks had been detected and isolated. Please be aware that this does not affect our EQX PoP directly, as the FR4 datacenter is unaffected. Temperatures were already checked manually overnight and found to be optimal. For the time being, the situation is as follows: - Our GTT ports at DRT-FRA8 are still working. We always operate our transit ports at both PoPs. Other transit providers (Tata, NTT) are unaffected. - Our transit capacity is reduced from 4.8 Tbps to 3.2 Tbps, which is still more than sufficient even for large-scale DDoS attacks. Also, multiple Tbps of peering capacity are entirely unaffected. However, depending on the distribution of individual attacks, short-term bottlenecks are possible on single links. - As traffic to and from GTT is routed via DRT-FRA8, we are experiencing an (expected) imbalance on internal backbone links between our datacenters. We are closely monitoring all load parameters and will take action if needed. - As various network operators are affected by this issue, customers might see slightly increased latencies. This is not necessarily an issue within our control, but our support team is happy to check if you notice anything unusual. Once the GTT ports recover, traffic will automatically shift back to our EQX-FR4 PoP. This can cause brief reconnects for existing TCP and UDP sessions on IP addresses with active DDoS protection.

2. resolved Aug 15, 2026, 04:56 PM UTC

   The cooling outage at EQX-FR5 has been resolved, and GTT has brought their infrastructure fully back online. Over the past few hours, all of our 400G ports with GTT at EQX-FR4 sequentially recovered and returned to normal operation. Throughout the entire event, our redundant backbone architecture and automatic failover performed exactly as designed, absorbing the link losses without customer-facing impact.

Read the full incident report →

Started Aug 12, 2026, 11:54 PM UTC· resolved Aug 13, 2026, 01:46 PM UTC

1. identified Aug 12, 2026, 11:54 PM UTC

   At 1:39, a linecard in our router MX480.MC-FRA01 reported major errors, leading to minor packet loss and timeouts for roughly 20 seconds. At 1:43, our NOC was alerted to the errors. At 1:47, while debugging the issue, the linecard failed completely before we were able to gracefully drain and restart it, causing another 15 seconds of minor packet loss and timeouts, confirmed by both our external and internal monitoring. The datacenter is running stable since then but with reduced redundancy, as this linecard served 50% of our edge traffic. The remaining bandwidth is sufficient, however. We will keep you posted on the next steps.

2. identified Aug 13, 2026, 12:54 PM UTC

   Our technician picked up the replacement card from our spare parts inventory recently and has arrived at the datacenter. We do not expect any issues during the installation of the replacement part, but would like to point out the residual risk associated with work on core infrastructure.

3. resolved Aug 13, 2026, 01:46 PM UTC

   The new linecard took over production load 30 minutes ago and is stable. No issues occurred during installation.

Read the full incident report →

Started Aug 08, 2026, 12:00 PM UTC· resolved Aug 08, 2026, 05:57 PM UTC

1. investigating Aug 08, 2026, 03:00 PM UTC

   Starting 14:00 CET, we observed several BGP flaps on IPv6 as well as IPv4 impacting the same handover as 2 days ago. Based on the current information, we suspect issues within TATA FRA metro transport. Customers might have experienced brief disconnects during convergence. To mitigate the situation, the impacted port has been disabled gracefully. We do not expect further direct impact and are in continues communication with TATA to resolve the situation.

2. monitoring Aug 08, 2026, 03:03 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Aug 08, 2026, 03:04 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Aug 08, 2026, 05:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 08, 2026, 10:54 AM UTC· resolved Aug 08, 2026, 02:07 PM UTC

1. investigating Aug 08, 2026, 10:54 AM UTC

   We are currently investigating this issue.

2. identified Aug 08, 2026, 10:58 AM UTC

   The issue has been identified and a fix is being implemented.

3. identified Aug 08, 2026, 11:08 AM UTC

   vps-008 and it's VMs are up and running after an Emergency Reboot, vps-017 is still in progress

4. monitoring Aug 08, 2026, 11:13 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Aug 08, 2026, 02:07 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 26, 2026, 03:24 PM UTC· resolved Jul 26, 2026, 03:35 PM UTC

1. investigating Jul 26, 2026, 03:32 PM UTC

   We are currently investigating this issue.

2. resolved Jul 26, 2026, 03:35 PM UTC

   All VMs are booted already, and the host system seems to be stable again.

Read the full incident report →

Started Jul 02, 2026, 09:24 PM UTC· resolved Jul 03, 2026, 04:44 AM UTC

1. investigating Jul 02, 2026, 09:24 PM UTC

   We are currently investigating this issue.

2. investigating Jul 02, 2026, 09:28 PM UTC

   We just performed a emergency reboot for this node and continue investigating the issue.

3. investigating Jul 02, 2026, 09:43 PM UTC

   The emergency reboot is done and we start to boot every seed on this node.

4. investigating Jul 02, 2026, 09:58 PM UTC

   The system is operational but we keep investigate the root cause

5. monitoring Jul 02, 2026, 10:01 PM UTC

   A fix has been implemented and we are monitoring the results.

6. identified Jul 03, 2026, 03:40 AM UTC

   The issue persists, thus we migrate all Seeds to another node.

7. monitoring Jul 03, 2026, 04:03 AM UTC

   A fix has been implemented and we are monitoring the results.

8. resolved Jul 03, 2026, 04:44 AM UTC

   All Seeds have been live migrated (without downtime) to another node. We keep investigating the root cause internally.

Read the full incident report →

Started Jul 01, 2026, 09:15 AM UTC· resolved Jul 01, 2026, 10:15 AM UTC

1. investigating Jul 01, 2026, 09:41 AM UTC

   We are currently investigating this issue.

2. identified Jul 01, 2026, 09:49 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 01, 2026, 09:54 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 01, 2026, 10:15 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 22, 2026, 01:25 AM UTC· resolved Jun 22, 2026, 01:45 PM UTC

1. investigating Jun 22, 2026, 01:25 AM UTC

   Between 03:03 and 03:14, packet loss was observed in parts of our network. Our team is currently investigating the situation.

2. monitoring Jun 22, 2026, 01:52 AM UTC

   Internal backbone links between our Edge Routing POPs (EQX-FR4, DRT-FRA8) were saturated with DDoS traffic. As very little traffic traverses these links, the impact was low. We have applied a fix for this situation, and it has not reoccurred since our initial status post.

3. resolved Jun 22, 2026, 01:45 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 21, 2026, 07:00 AM UTC· resolved Apr 23, 2026, 02:00 PM UTC

1. identified Apr 21, 2026, 08:43 AM UTC

   During routine maintenance of the UPS array for the A Feed, a malfunction occurred in one of the UPS modules. As a result, the system switched to bypass mode (automatically as intended), and the A Feed is currently running on emergency power. There was no impact on the IT infrastructure at any time. The datacenter operator is working with the the manufacturer to resolve the issue as quickly as possible and will switch back to normal operation once the issue is resolved. Till then, power on the A feed is not redundant and depending on emergency power. We will keep you updated on the progress.

2. monitoring Apr 21, 2026, 12:57 PM UTC

   The load on Feed A is back under UPS protection; power is currently being supplied via generator and will be switched back to the utility grid shortly.

3. monitoring Apr 21, 2026, 04:57 PM UTC

   The A-Feed is now back on municipal grid power and remains UPS-protected, as reported previously. Due to a faulty module that caused today's issue, the UPS cluster for the A-Feed is currently operating with reduced redundancy (N instead of N+1). Repair is scheduled for tomorrow. We will keep you updated.

4. resolved Apr 25, 2026, 08:13 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 30, 2026, 01:35 PM UTC· resolved Mar 30, 2026, 02:00 PM UTC

1. investigating Mar 30, 2026, 01:39 PM UTC

   We are currently investigating this issue.

2. monitoring Mar 30, 2026, 01:54 PM UTC

   The affected host is currently in recovery, and all VMs are starting. We will continue to monitor the situation.

3. resolved Mar 30, 2026, 02:00 PM UTC

   All VMs are booted already, and the host system seems to be stable again.

Read the full incident report →

Started Feb 28, 2026, 10:19 AM UTC· resolved Feb 28, 2026, 11:23 AM UTC

1. investigating Feb 28, 2026, 10:31 AM UTC

   We are currently experiencing an outage with the vServer node epyc9754-4 and the ts15 instance. Our team is actively working to resolve this issue. You can check in the control panel if you have VMs on the affected host.

2. monitoring Feb 28, 2026, 10:39 AM UTC

   The affected host is currently in recovery and all VMs are starting. We will continue to monitor the situation.

3. resolved Feb 28, 2026, 11:23 AM UTC

   All VMs are booted already and the host system seems to be stable again.

Read the full incident report →

Started Feb 26, 2026, 02:53 PM UTC· resolved Feb 26, 2026, 04:29 PM UTC

1. monitoring Feb 26, 2026, 02:53 PM UTC

   Between 15:37 CET and 15:46 CET, there were multiple instances of packet loss due to an issue with one of our upstreams. The dataforest network itself was not affected directly. Traffic flowing through this upstream was affected negatively till we cut the BGP sessions at 15:47. Connectivity has been stable since, and we continue to monitor the situation closely.

2. resolved Feb 26, 2026, 04:29 PM UTC

   Since our upstream fixed the issue, we have reestablished our BGP sessions with them 20 minutes ago, restoring full capacity and optimal routing of the dataforest network. More information about the incident can be found on their status page: https://status.gcore.com/cmm3me3mu0039xj9ljhuyyayt

Read the full incident report →

Started Feb 21, 2026, 09:10 AM UTC· resolved Feb 21, 2026, 09:45 PM UTC

1. identified Feb 21, 2026, 09:10 AM UTC

   We detected reachability issues for the prefixes 62.192.153.0/24 and 193.46.81.0/24 due to heavy DDoS attacks. Servers within these IP ranges were affected multiple times for a few minutes: 62.192.153.0/24 - 09:35 CET 193.46.81.0/24 - 9:35, 9:49, 10:01 CET Other customers not directly under attack may have noticed packet loss around these timestamps for a couple of seconds. We have implemented countermeasures and are working on further ones. Currently, the situation is stable.

2. resolved Feb 21, 2026, 09:45 PM UTC

   The situation remained stable, we are closing this incident.

Read the full incident report →

Started Feb 20, 2026, 04:27 PM UTC· resolved Feb 21, 2026, 06:34 PM UTC

1. identified Feb 20, 2026, 04:34 PM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Feb 20, 2026, 04:48 PM UTC

   At 17:12 and 17:16, our monitoring systems alerted us about packet loss via our DTAG port. At 17:24, 17:27, and 17:31, our monitoring systems alerted us about packet loss via our DE-CIX port. Investigation showed high incoming traffic congesting these ports. We have immediately disabled our sessions with these providers for now and are analyzing the root cause to implement appropriate measures. Only traffic entering our network via DTAG or DE-CIX was affected by this issue, no other paths / uplinks were affected.

3. resolved Feb 21, 2026, 06:34 PM UTC

   We are closing this incident, as the situation has been stable since this status post was created. The DE-CIX engineering team is still investigating yesterday's congestion affecting our peering port. Until further clarification, selected critical peering sessions remain disabled. Therefore we do not anticipate further issues with our DE-CIX peering. Inbound DTAG (Deutsche Telekom AG) traffic has been rerouted and will remain so until further notice to avoid similar situations. We will re-enable the remaining peering sessions once the root cause has been fully clarified with the DE-CIX team and the mitigation is in place. Only inbound traffic entering our network via DTAG or DE-CIX was affected for a few minutes till we disabled the sessions. All other uplinks/paths were not impacted. Outbound traffic was not affected.

Read the full incident report →

Started Feb 19, 2026, 08:49 PM UTC· resolved Feb 20, 2026, 04:35 PM UTC

1. monitoring Feb 19, 2026, 08:49 PM UTC

   We detected reachability issues for the prefix 45.157.234.0/24 due to heavy DDoS attacks. Servers within this subnet were affected at the following times: 20:52 CET 21:03 CET 21:07 CET 21:09 CET 21:15 CET Other customers not directly under attack may have noticed packet loss around these timestamps for a couple of seconds. We have implemented countermeasures and are working on further ones. Currently, the situation is stable.

2. resolved Feb 20, 2026, 04:35 PM UTC

   The situation remained stable, we are closing this incident.

Read the full incident report →

Started Jan 27, 2026, 09:11 PM UTC· resolved Jan 27, 2026, 09:27 PM UTC

1. investigating Jan 27, 2026, 09:13 PM UTC

   We are currently investigating this issue. You can check in our control panels if you have products on the affected hosts.

2. monitoring Jan 27, 2026, 09:21 PM UTC

   A defective RAM module caused the host system vps-018 to fail. The host is accessible again and is currently starting up all VMs, most of which are already back online.

3. resolved Jan 27, 2026, 09:27 PM UTC

   Recovery of first VM: 22:19 Recovery of last VM: 22:25

Read the full incident report →

Started Jan 14, 2026, 01:04 PM UTC· resolved Jan 14, 2026, 06:33 PM UTC

1. investigating Jan 14, 2026, 01:11 PM UTC

   We are currently experiencing increased latency at our firstcolo FRA4 location in Frankfurt. There is no packet loss.

2. monitoring Jan 14, 2026, 01:17 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 14, 2026, 06:33 PM UTC

   Yesterday, some physical links at our Equinix PoP flapped. While these flaps caused no customer impact, they led to an imbalance between internal links connecting our new Equinix PoP and our core PoP at Interxion. Increased bandwidth usage during the day resulted in critical utilization on one port, which required immediate correction. During the initial corrective change, one port at our firstcolo FRA4 location became saturated at 14:03, causing increased latency for some customers (without packet loss). At 14:15, a further adjustment was applied, successfully and permanently resolving the issue. Additional configuration changes were implemented over the following hours to prevent similar imbalances within our new MPLS environment in the event of link flaps. We are still investigating the root cause of the link flaps at Equinix. As redundancy is fully in place and the balancing issue has been resolved, we do not expect any further impact.

Read the full incident report →

Started Jan 07, 2026, 08:45 PM UTC· resolved Jan 07, 2026, 09:29 PM UTC

1. identified Jan 07, 2026, 08:55 PM UTC

   The issue has been identified and a fix is being implemented. You can check in our control panels if you have products on the affected hosts.

2. monitoring Jan 07, 2026, 09:03 PM UTC

   The affected TS3 instances were booted on replacement hardware and are available again. Other VMs on the host epyc9374f-5 will remain affected for a bit longer while we apply a BIOS update. This host contains some VMs for our Avoro Power Rootservers. The update mitigates an urgent CPU issue that caused the outage.

3. monitoring Jan 07, 2026, 09:23 PM UTC

   epyc9374f-5 is back online. The BIOS update was completed successfully. The first VMs are already up, and the remaining VMs are starting one after another automatically. Please do not intervene manually. It may take up to 15 minutes until the last VM is running again.

4. resolved Jan 07, 2026, 09:29 PM UTC

   All VMs are up.

Read the full incident report →