Voice Services - Voice Quality Issues Reported (03-01-23)

Update timeline

1. investigating Jan 03, 2023, 04:08 PM UTC

   We are currently aware that a number of business customers along with customers contacting our Customer Service team are experiencing issues relating to call quality. Date and Time Incident Reported - 03/01/23 @ 14:30 Services Impacted - Giganet provided SIP Trunk in Telehouse North Area / Locations Impacted - Nationwide Latest Update - We are waiting for our voice partner to provide an update regarding next steps. As soon as we have more information we will provide a further update. We apologise for any inconvenience this may cause.

2. investigating Jan 03, 2023, 09:55 PM UTC

   Earlier, we announced emergency engineering works that would affect Giganet provided SIP Trunk services in Telehouse North between 21:00 - 21:30. These works have not resolved the call quality issues so we will continue to work with our supplier partners to investigate further. We sincerely apologise for the inconvenience caused. Next update due at 01:00 04/01/2023.

3. monitoring Jan 04, 2023, 12:57 AM UTC

   Service to the affected circuits was restored at 22:33 on 03/01/2023. We will continue to monitor for 24 hours before resolving this incident. Again, we sincerely apologise for the inconvenience this may have caused.

4. identified Jan 04, 2023, 09:23 AM UTC

   Unfortunately we are still experiencing issues with call quality and are continuing to work with our voice partner to resolve this issue. We will provide a further update as soon as we have more information. Apologies for any inconvenience this has caused.

5. identified Jan 04, 2023, 11:23 AM UTC

   We continue to work on resolving this issue at the highest priority with our supplier partners to find the resolution, however short periods of high latency are still being observed at our SBC (Session Border Controller). A further update will be provided once we have more information. We apologise for the inconvenience caused.

6. identified Jan 04, 2023, 02:45 PM UTC

   We are continuing to work on this issue at the highest priority with our suppliers and voice partners. As soon as we have more information we will provide a further update. Again, we apologise for the inconvenience caused.

7. identified Jan 04, 2023, 08:23 PM UTC

   We announced emergency engineering works that will affect Giganet provided SIP Trunk services in Telehouse North. Affecting: Voice Services Date & time of maintenance window: 04/01/2023 21:00 - 04/01/2023 21:30 Service Impact: Intermittent loss of voice services for up to 30 minutes during the maintenance window Reason for Outage - To resolve call quality / degraded service issues We apologise for any inconvenience this may cause. Next update due at 21:45 04/01/2023.

8. identified Jan 04, 2023, 09:39 PM UTC

   The emergency engineering works affecting Giganet provided SIP Trunk services in Telehouse North are now complete. We will continue to monitor these services overnight. We apologise for the inconvenience caused. Next update is due at 09:00 05/01/2023.

9. monitoring Jan 04, 2023, 09:41 PM UTC

   The emergency engineering works affecting Giganet provided SIP Trunk services in Telehouse North are now complete. We will continue to monitor these services overnight. We apologise for the inconvenience caused. Next update is due at 09:00 05/01/2023.

10. identified Jan 05, 2023, 09:15 AM UTC

    We are continuing to monitor voices services in collaboration with our voice partner following the emergency engineering work overnight. Initial observations and feedback is that voice services and call quality is still impacted by the issue. A further update will be provided this morning once we agree next steps with our voice partner to resolve this issue. We apologise for the inconvenience this is causing.

11. monitoring Jan 05, 2023, 03:18 PM UTC

    A fix has been implemented and we are currently monitoring voice services. We will continue to monitor affected services for the next 24 hours before resolving this issue. Should you continue to experience issues that you feel are attributed to this incident please email - [email protected] or phone - 0330 333 3006 Again, we apologise for any inconvenience this may have caused.

12. resolved Jan 06, 2023, 04:38 PM UTC

    This incident has now been closed following 24 hours of monitoring and no further related issues being reported. Again, we apologise for any inconvenience this may have caused.

13. postmortem Jan 09, 2023, 10:59 AM UTC

    ## **Reason for Outage** ‌ **Date of outage** – 3rd January 2023 **Duration** – Intermittent over 3 days **Services affected** – Business voice customers and Giganet phone systems. **Impact to services** – Degraded voice quality, calls dropping and packet loss ‌ **Overview** On the 3rd January 2023 we were advised by a number of business and internal customers that they were experiencing issues with incoming and outgoing call quality. Following investigation from our Voice and Network Engineering teams it was evident that the issue was affecting a number of business and internal teams. Following investigation throughout the day and working with our voice partner it was suggested that we reboot the server which would in turn restart a number of services. A change was carried out at 21:00 on the 3/1/23, however this did not resolve the issue. On the 4/1/23 further analysis was carried out by Technical Teams and our voice partner. The Technical Teams identified sporadic spikes in CPU activity. Having discussed this issue with our voice partner it was suggested that we add additional processing power \(CPU\) and memory to the affected server. A change was therefore raised to increase CPU and Memory at 21: 00 on the 4/1/23. This change was carried out successfully, however this did not unfortunately fix the root cause. On the morning of 5/1/23 a failed disk was replaced in a SAN \(Storage Attached Network\). Following the replacement of the disk, packet loss and degraded call quality reduced significantly. The replacement disk / array finished rebuilding at 13:50 on 5/1/23 and following an hour monitoring services we provided an update on our Status Page advising that a fix had been implemented. **Resolution Actions** Server and processes rebooted / restarted CPU and Memory increased on Server SAN Disk Replaced **Root Cause Analysis** Failed SAN Disk in Telehouse North causing high CPU usage **Preventative Actions** Root cause has been captured. Failed SAN issue was not attributed to this issue, however knowledge base article created to allow quicker resolution should this happen again. Internal Teams are working on internal processes as part of Continual Improvement _PRB0040115_ _09-01-2023_