Vorco NZ experienced a minor incident on November 17, 2023 affecting Core Network and Internet & WAN Access, lasting 2h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 17, 2023, 02:00 AM UTC
Start Time: 1434 Service(s) Affected: International internet connectivity Customer Impact: Customers may be unable to reach websites and services hosted internationally, particularly those that route via the USA. This appears to be a fault upstream and we are liaising with our upstream transit for more information. Connectivity to Australia appears to be working still. We apologise for any inconvenience this may cause.
- identified Nov 17, 2023, 02:30 AM UTC
We've confirmed this is due to an upstream transit fault which for the time being has resolved itself. We're awaiting confirmation on whether a permanent fix is in place and will update this incident when we receive more details.
- resolved Nov 17, 2023, 03:44 AM UTC
This was resolved at 3:04pm. We'll update this incident with a root cause when available.
- postmortem Nov 23, 2023, 02:52 AM UTC
We have today received an RFO from our upstream transit provider. The cause of this incident was an upstream international transit outage between AS9790 and AS4826 caused by AS9790 bringing a new BGP peer via AS6939 online. Upon AS9790 bringing that new peer with AS6939 online, additional BGP routes got re-advertised by AS9790 to AS4826 resulting in AS9790 prefix limits with AS4826 being reached and accordingly BGP sessions automatically shut down. The impact of this was the loss of about 18% of international transit. As soon as the prefix limits were noticed the change was rolled back to restore connectivity and a post-incident review has been conducted to identify and address opportunities for improvement.