Penguin Internet experienced a minor incident on October 10, 2025, lasting 4d 12h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Oct 10, 2025, 11:30 AM UTC
Our monitoring systems have alerted us to a power supply failure on one of our core racks. The majority of servers in this rack are dual fed and so will continue to operate, however some less essential services may be affected. We are currently investigating the cause of this with the datacentre
- investigating Oct 10, 2025, 12:01 PM UTC
The datacentre are still investigating this. We saw power restored to the one APC and so services connected to that came back up, however we are seeing issues in another rack now, leading us to believe there is a power distribution issue at the datacentre, specifically within that particular data hall We will update as we receive an update from them.
- identified Oct 10, 2025, 12:57 PM UTC
The datacentre have confirmed that they have experienced power issues in the data hall that our affected racks are in, however at this point we do not know the cause of it. They have confirmed that the issue has been resolved, however at this point we are still investigating some further power distribution issues within our racks in that hall.
- identified Oct 10, 2025, 01:22 PM UTC
The datacentre have informed us that there has been a power outage to 4 data halls. Whilst the issue has been resolved, they have Schneider enroute to investigate further (Schneider manage the power infrastructure there). Whilst they have assured us that power has been restored, we are having issues with one of our PDUs which is indicating some damage to it following the incident. We are investigating this and have staff on their way to the datacentre too.
- monitoring Oct 10, 2025, 03:03 PM UTC
A small number of server customers were affected who did not have dual fed equipment and their servers are now back online again. We have suffered some PDU damage and will be replacing components, however this is for equipment that is now not currently now in use. We are therefore running at risk due to limited redundancy for those servers.
- resolved Oct 14, 2025, 11:30 PM UTC
We can confirm that the damaged power distribution equipment has been replaced and all infrastructure is running normally now