NZIX incident

[EMERGENCY OUTAGE] CHC-IX | CCE | pe1.chc1 Switch Lock Up

NZIX experienced a major incident on March 26, 2023 affecting CHC-IX, lasting 11m. The incident has been resolved; the full update timeline is below.

Started: Mar 26, 2023, 07:12 PM UTC
Resolved: Mar 26, 2023, 07:23 PM UTC
Duration: 11m
Detected by Pingoru: Mar 26, 2023, 07:12 PM UTC

Affected components

CHC-IX

Update timeline

identified Mar 26, 2023, 07:12 PM UTC

NZIX Engineers have been alerted to a management plane lock up on our switch at Christchurch Central Exchange (CCE) in CHC-IX. An emergency reboot is underway to resolve this. This bug is also impacting statistics on https://metrics.ix.nz/ for any service on this switch. Expected Downtime: ~5-10 minutes Summary of impact during this window: * Members connected to pe1.chc1 should expect their port to go offline. * ALL Members at CHC-IX should expect connectivity to rs1.chc.ix.nz to go down Due to the nature of the software fault, we cannot appropriately install BCP214 compliant filters to ports on the switch to reduce BGP disruption. This means ALL members that have routes via peers on pe1.chc1 may see a period of packet loss whilst BGP re-converges to alternate paths.
resolved Mar 26, 2023, 07:23 PM UTC

This incident has been resolved, newer firmware has been loaded that resolves this bug. Maintenance notices will be booked in soon to apply this code across all NZIX.