Resolved after 2 days 3 hours, on May 17, 2025.
Status 2025-05-19: We are currently working on the affected machine, resulting in downtime for Codeberg Pages, Hosted Woodpecker CI, Weblate/Codeberg Translate and meta pages (blog, docs, ...). This might take a while. One of our machines is operating at degraded performance due to hardware failure (an SSD disappeared from the RAID). We have moved Forgejo to another 'mountain' (we name our servers after mountains). Some secondary services remain on the affected machine and might operate with degraded performance (Codeberg Pages, Hosted Woodpecker CI, Weblate/Codeberg Translate). We have had issues with an SSD in that slot some months ago, where it was suddenly gone, but reappeared after a server reboot. Some weeks ago, the issue occured again, but the SSD was entirely dead. We replaced it earlier this week, but only after a few days of operation, the replacement disappeared again. We suspect a hardware issue with the nvme slot, which unfortunately also seems to entirely kill the drives that are inserted into it.
Update timeline
- investigating May 17, 2025, 09:47 AM UTC
Status 2025-05-19: We are currently working on the affected machine, resulting in downtime for Codeberg Pages, Hosted Woodpecker CI, Weblate/Codeberg Translate and meta pages (blog, docs, ...). This might take a while. One of our machines is operating at degraded performance due to hardware failure (an SSD disappeared from the RAID). We have moved Forgejo to another 'mountain' (we name our servers after mountains). Some secondary services remain on the affected machine and might operate with degraded performance (Codeberg Pages, Hosted Woodpecker CI, Weblate/Codeberg Translate). We have had issues with an SSD in that slot some months ago, where it was suddenly gone, but reappeared after a server reboot. Some weeks ago, the issue occured again, but the SSD was entirely dead. We replaced it earlier this week, but only after a few days of operation, the replacement disappeared again. We suspect a hardware issue with the nvme slot, which unfortunately also seems to entirely kill the drives that are inserted into it.
Your product runs on more than Codeberg
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services