Codeberg incident

Unexpected downtime

Resolved after 1 hour 29 minutes, on April 16, 2025.

investigating Apr 16, 2025, 01:39 PM UTC

Codeberg is currently suffering from unexpected downtime due to human error. We're restoring service as soon as possible. **Update 14.11 UTC:** It looks like one of the root nvme of our primary server went missing. Your data is safe due to RAID1 and/or clustered services. However, several services like Forgejo are still single-instance and only run on that specific machine. We're currently recovering the system disk. **Update 15.00:** We have booted the machine in a degraded state. However, we're having struggle synchronizing our Galera / MariaDB database cluster. The node refuses to rejoin the cluster.

Started
Apr 16, 2025, 01:39 PM UTC
Resolved
Apr 16, 2025, 03:09 PM UTC
Duration
1h 29m
Detected by Pingoru
Apr 16, 2025, 01:39 PM UTC

Update timeline

  1. investigating Apr 16, 2025, 01:39 PM UTC

    Codeberg is currently suffering from unexpected downtime due to human error. We're restoring service as soon as possible. **Update 14.11 UTC:** It looks like one of the root nvme of our primary server went missing. Your data is safe due to RAID1 and/or clustered services. However, several services like Forgejo are still single-instance and only run on that specific machine. We're currently recovering the system disk. **Update 15.00:** We have booted the machine in a degraded state. However, we're having struggle synchronizing our Galera / MariaDB database cluster. The node refuses to rejoin the cluster.

Your product runs on more than Codeberg

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →