I.T Communications Limited experienced a critical incident on January 13, 2025 affecting Outbound Calling and Volta Juniper MX Router and 1 more component, lasting 3h 35m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 13, 2025, 01:30 PM UTC
We have lost one of our primary routers at Volta Data Centre, We have applied workarounds to most services, with the exception of Leased Lines and DSL Connections. Phil is going to site to investigate the router issue.
- identified Jan 13, 2025, 02:08 PM UTC
We are still waiting for Phil to arrive at the Data Centre to investigate the issue. SIP and PBX services still seem to be affected despite moving the gateways to the other router. It seems our network cannot see the Google DNS Servers, We have applied another workaround to SIP Trunk servers to use our own DNS Servers which seems to be resolving ok. More tests to be done to try and bring up services while Phil fixes the Primary Router issue.
- identified Jan 13, 2025, 02:16 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 13, 2025, 02:25 PM UTC
Phil has now arrived at the Data Centre.
- identified Jan 13, 2025, 02:39 PM UTC
Phil has checked the primary router and its Dead and will no longer boot up. Phil is now looking at using one of our spare-backup routers (Same Make/Model) which is designed so that we can restore services should we have a router fail. Phil will load the config from backup and put the spare router into service, We will then make a call with Juniper for them to fix the primary router. We do not yet have an ETA to when the new router will be online. Hopefully within an hour.
- identified Jan 13, 2025, 03:08 PM UTC
We are waiting for Juniper to transfer the licence to the cold spare router.
- identified Jan 13, 2025, 03:41 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 13, 2025, 03:42 PM UTC
We have now put in a final workaround which will restore all SIP services,. while we continue to wait for Juniper to transfer the licence to the new router.
- identified Jan 13, 2025, 03:48 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 13, 2025, 04:29 PM UTC
The new router is now doing a reboot to apply config.
- monitoring Jan 13, 2025, 04:57 PM UTC
The replacement router is now live and in service and leased lines are now up.
- resolved Jan 13, 2025, 05:05 PM UTC
Now that we are up and running with the replacement router, we shall continue to to work with Juniper Networks to fix the failed router. We may at some point need to revert back to the original router or new replacement depending what we are required to do by Juniper. We will open a Maintenance window out of hours to do any work that would result in an outage. For now, full service is up and working.