Codeberg Outage History
Codeberg is up right nowPingoru has recorded 66 incidents for Codeberg: 9 in the last 12 months and 0 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 0 | 0 of 720 (0%) |
| May 2026 | 1 | 0 of 744 (0%) |
| June 2026 | 3 | 179 of 720 (25%) |
| July 2026 | 1 | 22 of 744 (3%) |
| August 2026 | 0 | 329 of 744 (44%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 0 | 0 of 744 (0%) |
| November 2026 | 0 | 0 of 48 (0%) |
Incidents, newest first
Notice May 2, 2025 Email delivery delayed –
Started May 02, 2025, 02:20 AM UTC· resolved May 02, 2025, 02:20 AM UTC
- investigating May 02, 2025, 02:20 AM UTC
Email delivery is currently delayed. Please be patient.
Notice April 17, 2025 Network maintenance and migrations –
Started Apr 17, 2025, 11:59 AM UTC· resolved Apr 17, 2025, 11:59 AM UTC
- investigating Apr 17, 2025, 11:59 AM UTC
We are performing various iterative changes to our network setup including firewalls and reverse proxies within the next days. These could lead to brief interruptions of connections and unavailability of services. We apologize for the inconvenience. We are also experimenting with the usage of [Anubis](https://anubis.techaro.lol/) for protecting some routes against malicious scraping and automated traffic. Please report any issues you encounter. An inventory of ongoing and completed work: * ✅ Avoid NAT across containers in internal network. * ✅ Replace ufw with native firewall rules. * ✅ Clean up and separate reverse proxy configs. * ✅ Implement secondary services using failover IP address and Caddy. * ✅ Protect services using Anubis. * ✅ Convert HAproxy config to Caddy config (with rate-limiting and all the protections). * ✅ Migrate Codeberg.org to failover IP address and Caddy. We expect no prolonged interruption due to this work.
Minor April 16, 2025 Unexpected downtime 1h 29m
Started Apr 16, 2025, 01:39 PM UTC· resolved Apr 16, 2025, 03:09 PM UTC
- investigating Apr 16, 2025, 01:39 PM UTC
Codeberg is currently suffering from unexpected downtime due to human error. We're restoring service as soon as possible. **Update 14.11 UTC:** It looks like one of the root nvme of our primary server went missing. Your data is safe due to RAID1 and/or clustered services. However, several services like Forgejo are still single-instance and only run on that specific machine. We're currently recovering the system disk. **Update 15.00:** We have booted the machine in a degraded state. However, we're having struggle synchronizing our Galera / MariaDB database cluster. The node refuses to rejoin the cluster.
Minor April 7, 2025 Codeberg inaccessible 3h 39m
Started Apr 07, 2025, 01:21 AM UTC· resolved Apr 07, 2025, 05:00 AM UTC
- investigating Apr 07, 2025, 01:21 AM UTC
05:00 UTC: Connectivity is slowly being restored, please be patient if a service is not yet online. Codeberg will be inaccessible for the early morning (European time). We apologize for this inconvenience.
Minor April 2, 2025 Codeberg updating –
Started Apr 02, 2025, 09:57 PM UTC· resolved Apr 02, 2025, 09:57 PM UTC
- investigating Apr 02, 2025, 09:57 PM UTC
Codeberg.org will be down for a few minutes to deploy a performance patch
Notice March 30, 2025 Scheduled storage maintenance on Thursday, April 03 –
Started Mar 30, 2025, 04:58 PM UTC· resolved Mar 30, 2025, 04:58 PM UTC
- investigating Mar 30, 2025, 04:58 PM UTC
**Date and time:** Thursday 2025-04-03, staring 19.00 CEST / 17.00 UTC **Expected impact:** Performance degradation or periods of unexpected downtime. We will try to minimize the impact and will ideally complete the maintenance without any impact. We are installing new NVMe drives for our Ceph storage cluster, which involves rebooting two nodes. The expected impact is small or ideally not noticeable. However, it might be the reason for unexpected issues on our systems. Our hosted Forgejo Action runner (beta service) won't be available during this time.
Notice March 24, 2025 Codeberg down 10m
Started Mar 24, 2025, 07:21 PM UTC· resolved Mar 24, 2025, 07:31 PM UTC
- investigating Mar 24, 2025, 07:21 PM UTC
Update: We've deployed a fix and Codeberg should be back up. We are currently aware that Codeberg is not accessible. We are working on a fix.
Notice March 18, 2025 Database maintenance –
Started Mar 18, 2025, 07:52 PM UTC· resolved Mar 18, 2025, 07:52 PM UTC
- investigating Mar 18, 2025, 07:52 PM UTC
We are performing scheduled maintenance on our databases on Wednesday March 19, starting around 09.00 UTC. This will involve **a few minutes of scheduled downtime** of Codeberg.org and dependent services such as Codeberg Pages, hosted Woodpecker CI and hosted Forgejo Actions, because the database host is renamed and services need to be adjusted manually. Following the maintenance, there might be degraded performance. We are converting our single-instance MariaDB deployment to a Galera Cluster, and the initial synchronization will take a while.
Notice March 7, 2025 Limited access to Codeberg 53m
Started Mar 07, 2025, 03:04 PM UTC· resolved Mar 07, 2025, 03:57 PM UTC
- investigating Mar 07, 2025, 03:04 PM UTC
15:56 Codeberg should be back to usable performance levels. We are currently experiencing what seems to be a wave of crawlers hitting expensive-to-compute endpoints; this causes Codeberg to be very slow to the point of being able to access it. We are working on restoring it to a usable level.
Notice March 7, 2025 Codeberg downtime –
Started Mar 07, 2025, 01:52 PM UTC· resolved Mar 07, 2025, 01:52 PM UTC
- investigating Mar 07, 2025, 01:52 PM UTC
We are deploying extra monitoring to be able to better observe the high CPU usage, likely being caused by crawlers. Downtime should be a few minutes.
Notice February 22, 2025 Forgejo actions 2m
Started Feb 22, 2025, 03:31 PM UTC· resolved Feb 22, 2025, 03:33 PM UTC
- investigating Feb 22, 2025, 03:31 PM UTC
We are aware that the Forgejo actions runners offered by Codeberg are currently not online and not running CI jobs. We expect this to be fixed by the end of this day.
Notice February 19, 2025 Ceph storage cluster maintenance –
Started Feb 19, 2025, 12:49 PM UTC· resolved Feb 19, 2025, 12:49 PM UTC
- investigating Feb 19, 2025, 12:49 PM UTC
We are performing scheduled maintenance on our Ceph storage cluster, upgrading to the latest version. No interruption of production services is expected, however degraded performance might be possible.
Minor February 13, 2025 Network-level DDoS 1d 13h
Started Feb 13, 2025, 10:55 AM UTC· resolved Feb 15, 2025, 12:28 AM UTC
- investigating Feb 13, 2025, 10:55 AM UTC
Situation as of 2025-02-14, 23.56 UTC: Connectivity via IPv4 is stable again. All traffic to Codeberg is now behind a DDoS mitigation system, pending DNS propagation. [More information on the DDoS mitigation system](https://social.anoxinon.de/@Codeberg/114005053744584371). Situation as of 2025-02-14, 21.40 UTC: We are currently suffering from another network-level DDoS attack, connectivity via IPv4 is impacted. If you are able to use IPv6 you should be able to have good connectivity to Codeberg. Situation as of 2025-02-14, 07.33 UTC: SSH access via IPv4 has been fixed and all systems should be fully available. It was previously unavailable due to a configuration mistake in the firewall, which was overlooked at the time of setting it up because DNS propagation delay has made the tests run on another host. We are still monitoring closely. Situaton as of 23.20 UTC: Email delivery has been restored a few hours ago (also see the [announcement on Mastodon](https://social.anoxinon.de/@Codeberg/113998529178075152) for details about this decision). **We dropped most email that was generated today**. If you require someone's attention, please ping them again. If you tried to register, please sign in and re-send your activation email (or re-register your account in case it was pruned due to the pending activation). The situation is mostly stable, except for occassional hiccups that seem to originate from the typical abusive AI crawling. Due to the current workarounds in place (to protect against DDoS), our normal abuse detection for AI crawling doesn't work very well. Situation as of 18.12 UTC: IPv6 connectivity was restored a while ago. IPv4 now also works after we discovered a configuration issue that lead to increased false positive rates in our rate-limiting. Email delivery was still impacted. Situation as of 13.56 UTC: We have added multiple servers to DNS entries. If you are lucky, you can use Codeberg just fine, depending on which machine is currently under attack. We are still investigating. We have been suffering from a network-level DoS that currently takes up all bandwidth available to our server. We are currently investigating potential workarounds to route traffic and restore connectivity.
Notice February 13, 2025 Email delivery interrupted –
Started Feb 13, 2025, 09:19 AM UTC· resolved Feb 13, 2025, 09:19 AM UTC
- investigating Feb 13, 2025, 09:19 AM UTC
We are currently investigating issues of interrupted email delivery.
Major January 23, 2025 System reboot –
Started Jan 23, 2025, 01:05 PM UTC· resolved Jan 23, 2025, 01:05 PM UTC
- investigating Jan 23, 2025, 01:05 PM UTC
We inadvertently froze our primary instance and decided to do a full reboot. We'll be back in a few minutes.
Notice December 23, 2024 Ceph storage maintenance 6h 1m
Started Dec 23, 2024, 08:38 PM UTC· resolved Dec 24, 2024, 02:39 AM UTC
- investigating Dec 23, 2024, 08:38 PM UTC
Huge steps ahead of us tonight: We are performing maintenance on our Ceph storage cluster and finally distributing data across several nodes to benefit from redundancy and fault tolerance features. We expect no service interruption but decreased performance while the storage is backfilling on the new nodes. Update: The storage distribution is in progress and is expected to take some a several hours to complete.
Notice December 4, 2024 Ceph cluster maintenance –
Started Dec 04, 2024, 06:22 PM UTC· resolved Dec 04, 2024, 06:22 PM UTC
- investigating Dec 04, 2024, 06:22 PM UTC
We are performing maintenance on our Ceph cluster, which might lead to **reduced performance or short periods of unavailability** in a few hours, **starting 20.00 UTC.** We are trying to minimize the impact on the systems. Work started yesterday and went well except for a brief interruption. Today, we are upgrading the Ceph version, which ideally also completes without downtime or interruption.
Notice November 20, 2024 Cron jobs filling CI queue overnight –
Started Nov 20, 2024, 01:35 AM UTC· resolved Nov 20, 2024, 01:35 AM UTC
- investigating Nov 20, 2024, 01:35 AM UTC
We have extended our monitoring for the hosted Woodpecker CI, including a check that jobs execute regularly and in time. Currently, cron jobs which run at about 0 UTC can fill the queue and report the CI as down, while it is just processing the backlog. We'll be adjusting our monitoring to reflect this.
Notice November 9, 2024 Scheduled reboot on Tuesday, November 12, 08.00 UTC –
Started Nov 09, 2024, 09:44 PM UTC· resolved Nov 09, 2024, 09:44 PM UTC
- investigating Nov 09, 2024, 09:44 PM UTC
Due to a hardware reboot, access to Codeberg.org and all services will be interrupted for about thirty minutes at 09.00 UTC (08.00 UTC), hopefully less. The reboot is required to apply some network-related configs and proceed with our new clustering setup. We hope it will be the last time this is necessary - but we actually promised this the last time already, so ... Thank you for your patience! We apologize for interrupting your work.