Is Vetta Online down?

During the period of ~2:00am ~ 8:35am this morning (17/09), some cloud compute customers with servers in Auckland and public IPv4 addressing contained within 103.85.28.0/24 subnet may have experienced degraded, or no IPv4 connectivity to these IP addresses. This was caused by this specific route not being distributed within our IGP following planned network maintenance overnight. Root cause investigation is underway, as well as increased monitoring of this IP subnet to proactively alert us to this issue in future. We apologise for any inconvenience caused. This issue was opened retrospectively.

Resolved

Some customers may be experiencing reduced connectivity, or higher latency to some destinations as a result of an incident with a core network element and subsequent routing path changes. Engineering teams are involved and investigating. Updates to follow.

Current investigations are showing the loss of two core network paths out of one of our core Auckland network PoPs. Normal traffic levels have moved to alternate and less-preferable traffic paths. Investigations are continuing.

Disrupted paths have been restored by the fibre vendor, and traffic has reconverged back to normal network paths. Our team is continuing to monitor the situation closely.

Primary paths have remained stable with normal traffic flow and this incident is now resolved. Our infrastructure team will continue communication with appropriate teams of the fibre vendors involved to identify root cause. We apologise for any inconvenience caused while traffic moved to alternate paths during this incident.

A connectivity disruption is currently being experienced to some hosted customer cloud servers, and our team is investigating.

Connectivity has been restored and the team are monitoring.

Incident has been resolved and further preventative measures have been taken. Maintenance will be announced soon to complete changes in response to this incident and its subsequent root cause investigation. We apologise for any inconvenience caused.

We are aware of impact to backend services supporting authentication of connections, which may impact new connections or DHCP renewals, impacting connectivity. Our team are investigating as a top priority. Updates to follow.

Backend systems are now recovering, and systems are under monitoring.

This incident is now resolved. Root cause investigation is underway. We apologise for any invonvenience caused.

From approx 1:50pm~2:20pm, Auckland cloud server customers may have experienced an impact to reachability of servers due to a systems issue. This is now resolved, and we apologise for any inconvenience caused. Root cause investigation is underway. This issue was opened retrospectively.

Resolved