Codeberg incident

Network-level DDoS

Resolved after 1 day 13 hours, on February 13, 2025.

investigating Feb 13, 2025, 10:55 AM UTC

Situation as of 2025-02-14, 23.56 UTC: Connectivity via IPv4 is stable again. All traffic to Codeberg is now behind a DDoS mitigation system, pending DNS propagation. [More information on the DDoS mitigation system](https://social.anoxinon.de/@Codeberg/114005053744584371). Situation as of 2025-02-14, 21.40 UTC: We are currently suffering from another network-level DDoS attack, connectivity via IPv4 is impacted. If you are able to use IPv6 you should be able to have good connectivity to Codeberg. Situation as of 2025-02-14, 07.33 UTC: SSH access via IPv4 has been fixed and all systems should be fully available. It was previously unavailable due to a configuration mistake in the firewall, which was overlooked at the time of setting it up because DNS propagation delay has made the tests run on another host. We are still monitoring closely. Situaton as of 23.20 UTC: Email delivery has been restored a few hours ago (also see the [announcement on Mastodon](https://social.anoxinon.de/@Codeberg/113998529178075152) for details about this decision). **We dropped most email that was generated today**. If you require someone's attention, please ping them again. If you tried to register, please sign in and re-send your activation email (or re-register your account in case it was pruned due to the pending activation). The situation is mostly stable, except for occassional hiccups that seem to originate from the typical abusive AI crawling. Due to the current workarounds in place (to protect against DDoS), our normal abuse detection for AI crawling doesn't work very well. Situation as of 18.12 UTC: IPv6 connectivity was restored a while ago. IPv4 now also works after we discovered a configuration issue that lead to increased false positive rates in our rate-limiting. Email delivery was still impacted. Situation as of 13.56 UTC: We have added multiple servers to DNS entries. If you are lucky, you can use Codeberg just fine, depending on which machine is currently under attack. We are still investigating. We have been suffering from a network-level DoS that currently takes up all bandwidth available to our server. We are currently investigating potential workarounds to route traffic and restore connectivity.

Started
Feb 13, 2025, 10:55 AM UTC
Resolved
Feb 15, 2025, 12:28 AM UTC
Duration
1d 13h
Detected by Pingoru
Feb 13, 2025, 10:55 AM UTC

Update timeline

  1. investigating Feb 13, 2025, 10:55 AM UTC

    Situation as of 2025-02-14, 23.56 UTC: Connectivity via IPv4 is stable again. All traffic to Codeberg is now behind a DDoS mitigation system, pending DNS propagation. [More information on the DDoS mitigation system](https://social.anoxinon.de/@Codeberg/114005053744584371). Situation as of 2025-02-14, 21.40 UTC: We are currently suffering from another network-level DDoS attack, connectivity via IPv4 is impacted. If you are able to use IPv6 you should be able to have good connectivity to Codeberg. Situation as of 2025-02-14, 07.33 UTC: SSH access via IPv4 has been fixed and all systems should be fully available. It was previously unavailable due to a configuration mistake in the firewall, which was overlooked at the time of setting it up because DNS propagation delay has made the tests run on another host. We are still monitoring closely. Situaton as of 23.20 UTC: Email delivery has been restored a few hours ago (also see the [announcement on Mastodon](https://social.anoxinon.de/@Codeberg/113998529178075152) for details about this decision). **We dropped most email that was generated today**. If you require someone's attention, please ping them again. If you tried to register, please sign in and re-send your activation email (or re-register your account in case it was pruned due to the pending activation). The situation is mostly stable, except for occassional hiccups that seem to originate from the typical abusive AI crawling. Due to the current workarounds in place (to protect against DDoS), our normal abuse detection for AI crawling doesn't work very well. Situation as of 18.12 UTC: IPv6 connectivity was restored a while ago. IPv4 now also works after we discovered a configuration issue that lead to increased false positive rates in our rate-limiting. Email delivery was still impacted. Situation as of 13.56 UTC: We have added multiple servers to DNS entries. If you are lucky, you can use Codeberg just fine, depending on which machine is currently under attack. We are still investigating. We have been suffering from a network-level DoS that currently takes up all bandwidth available to our server. We are currently investigating potential workarounds to route traffic and restore connectivity.

Your product runs on more than Codeberg

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →