Penguin Internet incident

Network Outage

Resolved after 5 hours 27 minutes, on December 9, 2025.

resolved Dec 09, 2025, 12:58 PM UTC

All services are now restored following the replacement of the router. Following the failure of one of our transit providers core routers, traffic should have failed over from them to the redundant link that they provide us with. It has been discovered that there is an issue with the failover process and we had no connectivity with their 2nd link. This is under investigation to identify the root cause of the matter, plus we will look at any network redesigns required to add further redundancy.

Started
Dec 09, 2025, 07:30 AM UTC
Resolved
Dec 09, 2025, 12:58 PM UTC
Duration
5h 27m
Detected by Pingoru
Dec 09, 2025, 07:30 AM UTC

Update timeline

  1. investigating Dec 09, 2025, 07:30 AM UTC

    We are currently investigating a major network outage at the datacentre. Currently no services are reachable, indicating a core network issue with our transit provider.

  2. investigating Dec 09, 2025, 07:41 AM UTC

    Our transit provider suspects a core router failure in the datacentre and have someone attending now to investigate. We will need to await their feedback, however will be questioning the redundancy failure from this incident once resolved. Please be assured we are doing all we can to restore service, however we have no control over the rectification time as the fault is not within our infrastructure.

  3. investigating Dec 09, 2025, 10:36 AM UTC

    The faulty hardware is in the process of being replaced, however the redundant failover from our transit provider has not taken effect, hence we are without any connectivity. They are investigating a routing issue to our equipment with a view to resolve this.

  4. resolved Dec 09, 2025, 12:58 PM UTC

    All services are now restored following the replacement of the router. Following the failure of one of our transit providers core routers, traffic should have failed over from them to the redundant link that they provide us with. It has been discovered that there is an issue with the failover process and we had no connectivity with their 2nd link. This is under investigation to identify the root cause of the matter, plus we will look at any network redesigns required to add further redundancy.

Your product runs on more than Penguin Internet

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →