Telesmart incident

Backhaul fault - NZ lower north island and south island

Major Resolved View vendor source →

Telesmart experienced a major incident on February 13, 2023 affecting NZ Datacentre - Datacom Kapua, Hamilton (KAP) and NZ Datacentre - Murphy Street, Thorndon, Wellington (ATT) and 1 more component, lasting 19d 23h. The incident has been resolved; the full update timeline is below.

Started
Feb 13, 2023, 08:08 PM UTC
Resolved
Mar 05, 2023, 07:28 PM UTC
Duration
19d 23h
Detected by Pingoru
Feb 13, 2023, 08:08 PM UTC

Affected components

NZ Datacentre - Datacom Kapua, Hamilton (KAP)NZ Datacentre - Murphy Street, Thorndon, Wellington (ATT)NZ Datacentre - Featherston Street, Wellington (CDC)NZ Datacentre - Enable Hornby, Christchurch (HNB)NZ Datacentre - Xtreme DC, Thorndon, Wellington (XDC)

Update timeline

  1. identified Feb 13, 2023, 08:08 PM UTC

    Telesmart would like to inform you that we are currently experiencing a weather-related event impacting network speeds south of Taupo. This is due to one of our backhaul link which traverses the east coast of the north island being down. Some customers can expect slower than normal speeds on our network until this issue is resolved. We have logged a fault with our backhaul provider. Estimated time of resolution: To be determined Next communication: 30 minutes

  2. identified Feb 13, 2023, 09:10 PM UTC

    Our technical team has identified the fault as a major fibre cut between Napier and Tarawera. This is impacting multiple carriers. ETR (estimated repair time) is not yet available. Due to Health and Safety reasons, contractors will not be able to send techs out in extreme weather conditions. They will dispatch a tech as soon as it is safe to do so. The weather situation is being closely monitored. Meanwhile we have made some network adjustments to lessen the impact and ease the load on diverse network backhaul. We are also investigating alternative backhaul options under emergency change conditions. Estimated time of resolution: TBA Next communication: 30 minutes

  3. identified Feb 14, 2023, 02:22 AM UTC

    Our backhaul provider has advised that due to several bridges being out in the area of the fibre cut that Downer techs are unable to reach the location to assess and repair. There is still no ETR at this stage. Meanwhile Telesmart is working on a rebuild of this backhaul using an alternate path.

  4. identified Feb 14, 2023, 07:16 PM UTC

    As of this morning the backhaul service in question remains down and work to restore the service is being hampered by weather, the national state or emergency and associated work restrictions triggered by that. We are continuing to work within our lines of communication with suppliers to try and get more information and a temporary replacement service in place.

  5. identified Feb 15, 2023, 12:58 AM UTC

    Our technical team has worked with suppliers on standing up temporary capacity which went live at 1:33pm today, we are now rebalancing the network to ease some congestion. While this will alleviate some congestion customers should still expect lower than normal performance until our high-speed optical backhaul is restored.

  6. identified Feb 17, 2023, 02:33 AM UTC

    Fibre techs are continuing to work on restoration. Several hundred meters of new overlay fibre optic cable has been run to resolve one of the many breaks in the fibre caused by landslides, debris and washed out bridges. Suppliers are hopeful of restoring some services to Gisbourne today through the northern fibre route but further fibre breaks elsewhere preclude the possibility of restoring our full backhaul service. ETR is still unknown at this stage.

  7. identified Feb 21, 2023, 08:52 PM UTC

    Technicians are continuing repair work on visible points of damage to the fibre between Tarawera and Bayview. Spark, Vodafone and 2Degrees are collaborating on the repair as they have all lost cables on similar paths in the area. ETA for repair is hopefully Friday or Saturday all going well.

  8. monitoring Feb 24, 2023, 02:54 AM UTC

    As at 15:38 today a fix has been implemented and services recovered on our high-speed optical backhaul. Vendor restoration confirmation has yet to be received so we will continue monitoring the situation while we wait for full repair confirmation.

  9. resolved Mar 05, 2023, 07:28 PM UTC

    Our technical team is happy that this issue is now completely resolved after a period of monitoring showed no further issues. If you continue to experience issues please contact our service desk on 0800 835 376 or [email protected]