Vorco NZ incident

Unplanned - Backhaul fault impacting Whangarei

Minor Resolved View vendor source →

Vorco NZ experienced a minor incident on May 21, 2018 affecting Internet & WAN Access, lasting 1h 50m. The incident has been resolved; the full update timeline is below.

Started
May 21, 2018, 08:00 PM UTC
Resolved
May 21, 2018, 09:50 PM UTC
Duration
1h 50m
Detected by Pingoru
May 21, 2018, 08:00 PM UTC

Affected components

Internet & WAN Access

Update timeline

  1. investigating May 21, 2018, 08:56 PM UTC

    We are aware of a third party backhaul fault impacting customers in Whangarei, we are working with our backhaul vendor to resolve this issue.

  2. identified May 21, 2018, 09:45 PM UTC

    We've confirmed the cause of the fault and an emergency change is being raised with our backhaul vendor to rectify this fault.

  3. resolved May 21, 2018, 09:50 PM UTC

    Traffic levels are now normal to/from Whangarei.

  4. postmortem Aug 03, 2018, 05:33 PM UTC

    This outage was caused by a migration within our backhaul vendor's network to retire an old router in Whangarei. The new router's configuration language differed from the old router and so after migration was completed our backhaul service no longer met the previous specifications. Once we became aware of this we raised it with our backhaul vendor to review and correct the backhaul specifications under an emergency change. We have queried this with our vendor as something that should have been checked immediately following the migration and asked that their processes be amended to reflect this.