I.T Communications Limited Outage History

Started Sep 21, 2026, 07:34 AM UTC· resolved Sep 21, 2026, 07:58 AM UTC

1. monitoring Sep 21, 2026, 07:34 AM UTC

   Hi, Our Support Desk "Ticketing System" provided by Zoho is currently running really slow.

2. resolved Sep 21, 2026, 07:58 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 03, 2025, 10:55 AM UTC· resolved Nov 03, 2025, 10:21 PM UTC

1. identified Nov 03, 2025, 10:55 AM UTC

   Hi, We are getting service issues with one of the servers in the Soft-Switch PBX Cluster Cl2 (PBX -03) We are going to give the server a full reboot as its been up for 299 Days. However there are a lot of active calls at the moment on PBX-CL2-03 so we need to wait for the call volumes to drop down.

2. identified Nov 03, 2025, 11:09 AM UTC

   We have completed a partial reboot of the server. If we still get notifications, we will do a full reboot.

3. identified Nov 03, 2025, 01:05 PM UTC

   We are still having issues with PBX-03, even after a full reboot. We have now escalated the issue internally to do further checks. Next Update at 13:30

4. identified Nov 03, 2025, 01:53 PM UTC

   We have found the cause to be a issue with one of the firewalls. We have put in a temp fix until this evening when the traffic levels are lower. so we can fully resolve the issue. Service should now be fully working.

5. resolved Nov 03, 2025, 10:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 03, 2025, 09:39 AM UTC· resolved Nov 03, 2025, 10:49 AM UTC

1. monitoring Nov 03, 2025, 09:39 AM UTC

   We are not able to access support tickets or our CRM today due to a issue with our CRM Provider "ZoHo" so their might be a delay in replying to tickets.

2. resolved Nov 03, 2025, 10:49 AM UTC

   Issue seem resolved.

Read the full incident report →

Started Jun 20, 2025, 12:50 PM UTC· resolved Jun 20, 2025, 10:35 PM UTC

1. investigating Jun 20, 2025, 12:50 PM UTC

   We have had reports that some customers are not able to reach our network. Initial investigations seem to suggest that the issue is within BT's Network. We are making contact with BT to find out more information.

2. identified Jun 20, 2025, 12:57 PM UTC

   Looks like a major issue with Cogent Communications, We are shutting down our BGP Sessions with Cogent Communications so traffic from BT will route via other routes.

3. identified Jun 20, 2025, 01:11 PM UTC

   BGP is now shutdown with the 3rd party network and traffic from BT is now reaching us via other routes.

4. monitoring Jun 20, 2025, 01:49 PM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring Jun 20, 2025, 02:13 PM UTC

   We have had some feedback from Cogent Communications "Customers in or routing through London, UK may be experiencing an outage . Cogent’s NOC is currently investigating the issue. There is no estimated time to repair. The master ticket is HD305149368" We shall leave the Cogent BGP Sessions down at both Sov and Volta until we get an all clear notification.

6. resolved Jun 20, 2025, 10:35 PM UTC

   Cogent have now fixed the fault and we have now restored the BGP Sessions. Next week we plan on on-boarding an additional Transit Provider at each location.

Read the full incident report →

Started May 14, 2025, 06:10 AM UTC· resolved May 16, 2025, 08:36 AM UTC

1. monitoring May 14, 2025, 06:10 AM UTC

   Microsoft are having issues with 365 Exchange Email forwarding. This is causing some emails to not reach our support desk. We have turned on Automatic Reply on our support tickets, If you do not get an automatic response, this means we may not have received your email. You can also submit tickets or provide updates to existing tickets directly by going to https://support.it-communicationsltd.co.uk/portal/en/signin

2. monitoring May 14, 2025, 06:13 AM UTC

   We are continuing to monitor for any further issues.

3. monitoring May 16, 2025, 08:31 AM UTC

   Looks like Microsoft has this morning fixed the issues with their exchange servers and our initial test shows we are now getting 100% of the emails sent to our support desk.

4. resolved May 16, 2025, 08:36 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 02, 2025, 03:12 PM UTC· resolved May 02, 2025, 03:22 PM UTC

1. investigating May 02, 2025, 03:12 PM UTC

   One of our carriers reported Intermittent issue with some inbound calls which is causing some calls to fail and not route to our network. Not all customers will be affected by this issue.

2. resolved May 02, 2025, 03:22 PM UTC

   The issue with inbound calls appears to be cleared.

Read the full incident report →

Started Apr 14, 2025, 12:53 PM UTC· resolved Apr 14, 2025, 05:28 PM UTC

1. investigating Apr 14, 2025, 12:53 PM UTC

   We have had reports of some websites not loading or not loading correctly www.argos.co.uk Barclays Online Banking bbc.co.uk We believe the issue relates to AS20940 Akamai Technologies, Inc. We are still investigating.

2. investigating Apr 14, 2025, 12:59 PM UTC

   The issue is not on I.T Communications network and therefore we have little to no control to when the issue will be resolved. AS20940 Akamai Technologies, Inc. is a CDN Network for many websites and it would seam part of their network are down. We are reaching out to Akamai Technologies to see if we can find out any more info.

3. investigating Apr 14, 2025, 01:21 PM UTC

   As a temp workaround we have shutdown the BGP Session for the MLP / Route Servers at Linx Now that AS20940 Akamai Technologies is taking a different network path, traffic is now working ok. We will submit a ticket with Akamai Technologies as it seems the issue is with Akamai Technologies over the Linx Network.

4. investigating Apr 14, 2025, 01:34 PM UTC

   We have now confirmed that Akamai hosted properties (Argos, BBC, Barclays etc) are working properly with the LINX RS sessions disabled. Akamai NOC have told me they have no known issues. Outgoing traffic is now currently going via Cogent Transit. Phil is trying to collect information on where the problem might exist, as, given involvement of the RS, it could potentially be a forwarding problem at LINX LON1 network.) While customers should no loner be having any issues, we shall continue to work on this until we have confirmed the issue to be resolved.

5. investigating Apr 14, 2025, 01:40 PM UTC

   We believe there is a forwarding problem on LINX LON1. We was observing loss for destinations behind Akamai, learned via the Route Servers. we cannot ping Akamai’s router IP, 195.66.224.168, from I.T Comms LON1 IP 195.66.226.251. At least one other network – Luminet – is reporting similar problems, and is unable to ping Akamai’s from their Slough LON1 port, but can from their HEX LD8 port. Dominykas Valys | NOC Engineer at Linx.net is now investigating the issue.

6. investigating Apr 14, 2025, 02:05 PM UTC

   We are continuing to investigate this issue.

7. investigating Apr 14, 2025, 02:07 PM UTC

   We have just had the below update from the London Linx Network Hi, We can confirm that there are no L2 issues at Akamais port at the moment. Their route server sessions have been flapping which has caused issues for multiple members on the LON1 LAN. We have raised a ticket with them to confirm if this is an ongoing maintenance/issue. We will keep you updated on their response but if you would like to reach out to them directly their NOC contacts are: [email protected] [email protected] Kind Regards, Dominykas Valys | NOC Engineer

8. resolved Apr 14, 2025, 05:28 PM UTC

   Hi, Akamai have confirmed that their port is not stable on the new router they have migrated to. They have since withdrawn their advertised routes from the route servers and we are seeing their port with no traversing traffic. Kind Regards, Dominykas Valys | NOC Engineer

Read the full incident report →

Started Feb 06, 2025, 06:15 PM UTC· resolved Feb 06, 2025, 11:19 PM UTC

1. investigating Feb 06, 2025, 06:15 PM UTC

   One of our 100G Fibre Wavelengths Between Volta Data Centre, 36-43 Great Sutton St, London EC1V 0AB and Sovereign House, 227 Marsh Wall, London E14 9SD Data Centres is triggering alarms on our monitoring systems for RX Frame Errors. As a precaution, we attended both locations today and swapped out the Fibre SFP Modules, to see if that cleared the errors. During the SFP Swap out, some active phone calls may have dropped / disconnected depending on how customers reach our network. There would have been two incidents where calls may have disconnected. Around 15:40 and 17:14 today. Unfortunately, the errors are still showing. We have therefore reported this to our wavelength provider to investigate.

2. identified Feb 06, 2025, 07:37 PM UTC

   Closer investigation has revealed that it seems likely our Wavelength provider "SSE Group" has a defective LR4 optic facing our equipment in Volta One (and only one) of the mux lanes has been transmitting gradually less and less power. We are now awaiting our Wavelength provider "SSE Group" to send someone out to Volta to replace a suspected LR4 Fibre Module failure. They have now agreed this is the likely cause and looking to dispatch an engineer to replace the LR4 Fibre Module.

3. identified Feb 06, 2025, 08:03 PM UTC

   Update from Our Wavelength provider part of the SSE Group. Field Engineer & part due onsite at Volta DC tonight at 23:00

4. resolved Feb 06, 2025, 11:19 PM UTC

   Our Wavelength provider part of the SSE Group has now replaced the faulty hardware and alerts on our monitoring is now clear.

Read the full incident report →

Started Jan 14, 2025, 09:49 AM UTC· resolved Jan 14, 2025, 10:01 AM UTC

1. investigating Jan 14, 2025, 09:49 AM UTC

   We are aware that some customers are not able to browse the internet. We believe that we have found the issue and putting in a fix to resolve. Update to folllow.

2. resolved Jan 14, 2025, 10:01 AM UTC

   Hi, The issue is now fully resolved.

Read the full incident report →

Started Jan 13, 2025, 01:30 PM UTC· resolved Jan 13, 2025, 05:05 PM UTC

1. identified Jan 13, 2025, 01:30 PM UTC

   We have lost one of our primary routers at Volta Data Centre, We have applied workarounds to most services, with the exception of Leased Lines and DSL Connections. Phil is going to site to investigate the router issue.

2. identified Jan 13, 2025, 02:08 PM UTC

   We are still waiting for Phil to arrive at the Data Centre to investigate the issue. SIP and PBX services still seem to be affected despite moving the gateways to the other router. It seems our network cannot see the Google DNS Servers, We have applied another workaround to SIP Trunk servers to use our own DNS Servers which seems to be resolving ok. More tests to be done to try and bring up services while Phil fixes the Primary Router issue.

3. identified Jan 13, 2025, 02:16 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Jan 13, 2025, 02:25 PM UTC

   Phil has now arrived at the Data Centre.

5. identified Jan 13, 2025, 02:39 PM UTC

   Phil has checked the primary router and its Dead and will no longer boot up. Phil is now looking at using one of our spare-backup routers (Same Make/Model) which is designed so that we can restore services should we have a router fail. Phil will load the config from backup and put the spare router into service, We will then make a call with Juniper for them to fix the primary router. We do not yet have an ETA to when the new router will be online. Hopefully within an hour.

6. identified Jan 13, 2025, 03:08 PM UTC

   We are waiting for Juniper to transfer the licence to the cold spare router.

7. identified Jan 13, 2025, 03:41 PM UTC

   We are continuing to work on a fix for this issue.

8. identified Jan 13, 2025, 03:42 PM UTC

   We have now put in a final workaround which will restore all SIP services,. while we continue to wait for Juniper to transfer the licence to the new router.

9. identified Jan 13, 2025, 03:48 PM UTC

   We are continuing to work on a fix for this issue.

10. identified Jan 13, 2025, 04:29 PM UTC

    The new router is now doing a reboot to apply config.

11. monitoring Jan 13, 2025, 04:57 PM UTC

    The replacement router is now live and in service and leased lines are now up.

12. resolved Jan 13, 2025, 05:05 PM UTC

    Now that we are up and running with the replacement router, we shall continue to to work with Juniper Networks to fix the failed router. We may at some point need to revert back to the original router or new replacement depending what we are required to do by Juniper. We will open a Maintenance window out of hours to do any work that would result in an outage. For now, full service is up and working.

Read the full incident report →

Started Dec 04, 2024, 12:27 AM UTC· resolved Dec 04, 2024, 04:57 AM UTC

1. investigating Dec 04, 2024, 12:27 AM UTC

   We are currently aware of a issue with BGP Flapping which is causing parts of our network to be unreachable. We are rebooting the affected Router to clear the fault.

2. investigating Dec 04, 2024, 12:34 AM UTC

   The Router has rebooted, however the BGP sessions are still having issues. We are investigating to find out the cause.

3. investigating Dec 04, 2024, 12:47 AM UTC

   We continue to investigate and are taking steps to resolve the issue.

4. identified Dec 04, 2024, 01:14 AM UTC

   Possible Cause of issue An Improper Handling of Exceptional Conditions vulnerability in the routing protocol daemon (RPD) of Juniper Networks Junos OS When a malformed BGP UPDATE packet is received over an established BGP session, RPD crashes and restarts. Continuous receipt of the malformed BGP UPDATE messages will create a sustained Denial of Service (DoS) condition for impacted devices. It would seem someone remotely is trying to cause a Denial of Service (DoS) attack on part of our network. Phil is now on his way to Sov House to load a software update on the affected router to hopefully resolve the issue.

5. identified Dec 04, 2024, 01:24 AM UTC

   Phil is expected to arrive on-site at 1:39am

6. identified Dec 04, 2024, 01:53 AM UTC

   Phil has now arrived onsite and will proceed to upgrading the Router Software, once updated and router is stable, Phil will attend second data centre and upgrade the software on the other Juniper MX Router. Update to follow.

7. identified Dec 04, 2024, 03:29 AM UTC

   Juniper Networks Junos OS upgrade finished and doing final reboots on 1st Router. Second Router can be upgraded from within the network so no need for Phil to go to second Data Centre. Network at Volta should start to become stable again very shortly.

8. identified Dec 04, 2024, 03:50 AM UTC

   Router is now stable.

9. identified Dec 04, 2024, 03:58 AM UTC

   The second router is also stable at the moment, However due to the BGP DDOS, we are going to do the upgrade now. We shall be starting the work shortly after 4:30am,

10. identified Dec 04, 2024, 04:32 AM UTC

    The new software is being loaded onto the second router now and will require a few reboots.

11. resolved Dec 04, 2024, 04:57 AM UTC

    last Router now backup and issue should now be fully resolved.

12. postmortem Dec 04, 2024, 11:11 AM UTC

    at around 12:00 am on 4th December 2024, our monitoring system alerted us to over 60 BGP sessions keep going up and down \(Flapping\) Our on-call network engineer Phil was then alerted and found that the BGP Daemon keep crashing and that is what caused the BGP sessions to keep flapping. A remote reboot of Sov Router did not resolve the issue and after investigating further, we found this to be likely a remote BGP DDOS Attack, which took advantage of a vulnerability in the routing protocol daemon \(RPD\) of Juniper Networks Junos OS \(Router\) Juniper released a fix for this which Phil attended both Data Centres to apply the software update. This could not be done remotely due to the BGP Daemon not being stable and a fear of software updating failing. So had to be done via a USB Drive to load on the software. Phil arrived at the first Data Centre at 1:53am of 4th December 2024 and attempted to load the new firmware. Following the update a few manual reboots was needed for the new software to fully apply. At 3:50am of 4th December 2024, the first Router then become stable again following the completed software upgrade. At 3:58 of 4th December 2024, we noticed the other router had also been stable for around 3 hours. Phil thought it best to also upgrade the software on the second router of which started at 4:32am of 4th December 2024. At 4:57am of 4th December 2024, the software had been fully upgraded and router rebooted and all the remaining BGP Sessions came up and no further alerts showing on our monitoring. Both upgrades fixed the vulnerability in the Juniper routing protocol daemon \(RPD\) and our network returned back to being stable. whilst it took from around 12:00am to 4:57am to fully resolve all the issues, the whole network would not have been completely down for this amount of time. **Affected Services** Hosted 3CX at Volta - around 3 Hours and 50 minutes Hosted 3CX at Sov House, few small outages but largely remained up and in service. SIP Trunks, Some customers would have been offline, depending on where the Customer 3CX was hosted, Internal network remained stable and fully functional during the Router issues,

Read the full incident report →

Started Nov 20, 2024, 10:44 AM UTC· resolved Nov 20, 2024, 11:45 AM UTC

1. investigating Nov 20, 2024, 10:44 AM UTC

   We have had reports of some calls are failing for both inbound and outbound. We are Investigating the issue and an update will be posted ASAP.

2. investigating Nov 20, 2024, 10:51 AM UTC

   We are aware of some issues that may be impacting calls – engineers are investigating and more information will follow as soon as possible.

3. identified Nov 20, 2024, 11:08 AM UTC

   Following on from the below notice - we are aware that customers have experienced issues with both inbound and outbound calls this morning. Engineers have located the cause and are in the process of fully restoring service - more updates will follow as soon as they are available. We apologise for the disruption.

4. identified Nov 20, 2024, 11:14 AM UTC

   We are still experiencing some issues with both inbound and outbound calls. Our engineers are in the process of fully restoring service - more updates will follow as soon as they are available. We apologise for the disruption.

5. resolved Nov 20, 2024, 11:45 AM UTC

   The issue affecting inbound and outbound calls this morning is now cleared. Our engineers are in the process of investigating and more details will follow as soon as we can. We sincerely apologise for the disruption this morning.

Read the full incident report →

Started Nov 07, 2024, 05:22 PM UTC· resolved Nov 07, 2024, 05:41 PM UTC

1. investigating Nov 07, 2024, 05:22 PM UTC

   We have lost one of our Power Feeds to our primary rack at Sovereign House. There should be no impact to customer services due to all our servers being dual power fed. We have made contact with the Data Centre and we are working with them to resolve the issue.

2. investigating Nov 07, 2024, 05:33 PM UTC

   Staff are now looking into the issue on the 2nd Floor.

3. resolved Nov 07, 2024, 05:41 PM UTC

   One of the power feeds had tripped. This has now been reset and full power restored.

Read the full incident report →