Hero Internet experienced a notice incident on April 3, 2023 affecting Voice and Broadband, lasting 1h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 03, 2023, 12:42 AM UTC
We are currently investigating an apparent network related issue where some networks are unreachable to/from the Hero network. Engineers are investigating the issue and looking at possible peering related issues to/from those networks. Further updates will be posted shortly to clarify the affected networks.
- identified Apr 03, 2023, 12:44 AM UTC
The issue appears to be a network problem/outage on the Auckland Peering Exchange: https://status.ix.nz/ From the AKL-IX status page: [OUTAGE] AKL-IX | Layer 2 reachabilitySubscribe Investigating - We are currently investigating reports of partial breaks in layer 2 communications to peers between MDR and Datacentre220.
- identified Apr 03, 2023, 01:01 AM UTC
Our engineers have attempted to route traffic away from the Auckland-IX peering Exchange to resolve the connectivity problems until the outage on AKL-IX has been resolved. Please contact our support team if you are still experiencing any connectivity problems. Hopefully by stopping advertising these routes via AKL-IX the traffic will route via a different route and bring connectivity back for those customers to our network. You can follow the status of the peering exchange fault at: https://status.ix.nz/
- monitoring Apr 03, 2023, 01:10 AM UTC
Our workaround re-routing Auckland-IX peering exchange traffic appears to have resolved the connectivity problem for those impacted customers. Moving this to Monitoring state, however if you are still having any problems reaching our network then please let our support team know immediately. We will set the status to resolved once AKL-IX have resolved the problem on their network and traffic has been re-established with AKL-IX. However, we believe that customers shouldn't be experiencing any problems currently with our work-around in place.
- resolved Apr 03, 2023, 02:40 AM UTC
AKL-IX UPDATE: A fix has been implemented as of 2:03pm (NZST). Impacts were seen between peers on pe2.akl3 at MDR and pe3.akl1 at Datacentre220 with the root cause still under analysis. Our engineers have confirmed that the AKL-IX peering is now fixed and we are able to connect with those peers previously affected by this outage on AKL-IX so we will begin moving traffic back to the AKL-IX peering exchange shortly and we are marking this event as resolved.