Hero Internet experienced a notice incident on March 5, 2025 affecting Broadband, lasting 1h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 05, 2025, 05:15 PM UTC
We are investigating a network problem on the Hero network. Updates will follow shortly
- investigating Mar 05, 2025, 06:16 PM UTC
Isolated to fibre broadband services via Chorus. Issue appears to be on Chorus side, and has been raised with them, and is under investigation.
- monitoring Mar 05, 2025, 06:40 PM UTC
As of a few minutes ago, we now see a normal level of Chorus connected services on line. Resolution steps were taken by Chorus, but we don't yet know what the original cause was.
- resolved Mar 05, 2025, 06:56 PM UTC
Broadband services are operating normally since just after 7:30.
- postmortem Mar 06, 2025, 08:22 PM UTC
Chorus have investigated and advised the reason identified for the handover outage early on the 6th of March. In summary, the day prior \(5th march\) there was a planned migration event within the Chorus network which involved this handover, which was completed successfully. A separate second planned migration on the 6th on the same Chorus network elements, which did not intend any handover changes was run, but which incorrectly rolled back or ‘undid’ aspects of previous days changes, leaving most \(but not quite all\) services on the handover inoperative. Essentially, the second change contained stale network configuration data, and Chorus are presently continuing to investigate specifically how that occurred, in order to prevent any recurrence. While Hero could not have prevented or reasonably predicted this particular event, we will also review how we can learn from this occurrence and our resolution steps. Due to how the fault was manifesting to us, some response time was spend first verifying our network and taking steps to isolate and confirm the issue was outside of our own equipment prior to escalating to Chorus.