Probado experienced a major incident on September 19, 2023 affecting VoIP (Voice over IP) and Data Services (Hosting), lasting 5d 18h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 19, 2023, 02:49 PM UTC
The issue has been identified and is currently with the Internet Service Provider. The issue has been classified as Congestion Loss of Connectivity. Updates will be relayed to us at least every hour.
- identified Sep 19, 2023, 03:40 PM UTC
Update from ISP @ 16.36: An issue has been identified with 9 BT bearers are down due to BT fibre issue. This has impacted Broadband traffic, traffic has diverted to resilient 9 other links via dynamic routing however customers may be experiencing some service degradation at this time Daisy Technical Lead has advised BT Openreach have been engaged to investigate. BT have deployed engineers to investigate but have not given ETAs at this time At this stage, no available workarounds are known to alleviate customer impact Further updates will follow as they become available.
- identified Sep 19, 2023, 05:23 PM UTC
Technical Lead has confirmed confirmed 9 BT bearers are down due to BT Openreach fibre issue. Core Network Operations has escalated this issue to BT Openreach Technical Lead has advised the next actions are for BT Openreach to locate the fibre fault. Technical Lead remains in contact with BT Openreach for regular updates We have had confirmation that the traffic will continue to redistribute via balancing scripts over the remaining bearers, this will reduce the congestion
- identified Sep 20, 2023, 03:02 PM UTC
Technical Lead has confirmed traffic normalising over the bearers and links restored. We are waiting on BT to confirm resolution.
- resolved Sep 25, 2023, 09:11 AM UTC
This incident has been resolved.