Intermittent Connectivity Issues affecting some GPON and Ethernet Business Services

Update timeline

1. investigating Oct 21, 2022, 09:07 AM UTC

   We are currently aware of an issue in the CityFibre network causing intermittent connectivity across GPON business services. Start Date/Time : 21/10/2022 - 3:45am Carrier: CityFibre Location: National CityFibre are currently investigating. As soon as we have more information we will provide a further update We apologise for any inconvenience this may cause.

2. investigating Oct 21, 2022, 10:56 AM UTC

   CityFibre have assigned this incident to their Network Operations Centre who are currently investigating the issue. As soon as we have any more information we will provide a further update. If you are experiencing any issues that you feel are attributed to this incident please contact [email protected]

3. investigating Oct 21, 2022, 12:05 PM UTC

   CityFibre have advised that they continue to investigate this issue. As soon as we have any more information we will provide a further update. If you are experiencing any issues that you feel are attributed to this incident please contact [email protected]

4. investigating Oct 21, 2022, 01:43 PM UTC

   CityFibre are still investigating the root cause that is causing intermittent connectivity across these services. We apologise for any inconvenience this may cause. As soon as we have any more information, we will provide a further update. If you are experiencing any issues that you feel are attributed to this incident, please contact [email protected]

5. investigating Oct 21, 2022, 04:01 PM UTC

   CityFibre continue to investigate the root cause that is causing intermittent connectivity across these services. We apologise for any inconvenience this may cause. As soon as we have any more information, we will provide a further update. If you are experiencing any issues that you feel are attributed to this incident, please contact [email protected]

6. investigating Oct 21, 2022, 05:50 PM UTC

   CityFibre continue to investigate, however, some services have been restored and are stable but others remain intermittent. If you are still having issues with your service, CityFibre recommend rebooting all kit in the interim while they continue their investigations. If you require any support to identify which equipment needs rebooting, please contact us on our business support line. We apologise for any inconvenience this may cause.

7. investigating Oct 21, 2022, 08:54 PM UTC

   CityFibre continues to investigate the root cause of the issue, however, some services have been restored and are stable but others remain intermittent. If you are still having issues with your service, CityFibre recommend rebooting all kit in the interim while they continue their investigations. If you are experiencing any issues that you feel are attributed to this incident please contact [email protected] Once again, we apologise for any inconvenience this may cause.

8. investigating Oct 22, 2022, 08:20 AM UTC

   CityFibre are still investigating the root cause that is causing intermittent connectivity across these services, however, some services have been restored and are stable but others remain intermittent. If you are still having issues with your service, CityFibre recommend rebooting all kit in the interim while they continue their investigations. If you are experiencing any issues that you feel are attributed to this incident please contact [email protected] As soon as we have any more information, we will provide a further update. We apologise for any inconvenience this may cause.

9. investigating Oct 22, 2022, 10:25 AM UTC

   Giganet have observed that all services have now been restored and are no longer seeing high packet loss. CityFibre are still investigating the root cause but we recommend that if you continue to have issues, you complete a full reboot of all equipment. If you are experiencing any issues that you feel are attributed to this incident please contact [email protected]. We apologise for any inconvenience this may cause.

10. investigating Oct 24, 2022, 08:37 AM UTC

    We have received further reports that some business customers are still experiencing intermittent connectivity issues. This has been escalated as a priority to our carrier (CityFibre) who are investigating this further. If you are experiencing an issue please complete a full reboot of all equipment as this may resolve this issue. Following a reboot of equipment, if you are still experiencing issues please contact [email protected]. We apologise for any inconvenience this may cause.

11. identified Oct 24, 2022, 01:45 PM UTC

    Our carrier (CityFibre) have advised that they have identified the cause of this issue and are looking to implement a change over the weekend of 29th / 30th October. This change will not be service impacting. If you continue to experience any issues you feel are attributed to this issue prior to the above change being implemented, please carry out a full reboot of your equipment. If you have any further questions relating to this please contact [email protected] We apologise for any inconvenience this causes.

12. monitoring Oct 31, 2022, 02:38 PM UTC

    Our partner carrier have confirmed that they have implemented the change over the weekend to address the root cause associated with this incident. We have now closed this incident following the update from our partner advising that the change was successful. Apologies for any inconvenience this may have caused.

13. resolved Oct 31, 2022, 02:38 PM UTC

    This incident has been resolved.

14. postmortem Nov 01, 2022, 04:06 PM UTC

    # **Reason for Outage** **Date of outage** – 21st October 2022 **Duration** – Intermittent over 3 days **Services affected** – Business Ethernet and GPON services **Impact to services** – Intermittent Connectivity Issues and Packet Loss **Overview** On the 21st October we were advised by a number of customers that they were experiencing intermittent connectivity issues. Following investigation from our Technical and Network Engineering teams it was evident that the issue was affecting a number of circuits nationwide on a partner carrier’s network. Our Technical Support team reported this incident to the carrier who arranged for their NOC \(Network Operations Centre\) to carry out further investigation. During the weekend of 22nd / 23rd October we attempted to contact our partner for updates, however these were not forthcoming hence the limited updates provided by us \(Giganet\). On Monday 24th October we initiated an urgent call with our partner to understand the present position and understand what was required to resolve the connection issues. It was suggested that a reboot of all equipment at customer sites should resolve the issue, however a change would be required by our partner on their network to increase stability going forward. **Resolution Actions** Customer equipment rebooted. Change implemented to improve LSP \(Link State Packet\) stability and fix a ‘bug’ **Root Cause Analysis** LSP instability Issues **Preventative Actions** Vendor has confirmed they will look at internal processes especially relating to communication channels to ensure regular updates are available at weekends Reference - PRB0040106 01-Nov-2022