NZIX experienced a major incident on March 31, 2023 affecting AKL-IX, lasting 1h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 31, 2023, 12:07 AM UTC
Our switch at VDC Albany crashed and is back online, investigations are underway.
- monitoring Mar 31, 2023, 12:42 AM UTC
The issue stemmed from an ephemeral event driver corruption, leading to a watchdog-triggered system reset. We are reaching out to our vendor for further examination, as recurring incidents point to hardware failure that warrants an RMA. We are currently monitoring the situation while gathering and analysing debug logs.
- resolved Mar 31, 2023, 01:41 AM UTC
The findings remain indeterminate, with a transient issue being the tentative cause. Nevertheless, we have cold spare hardware at the ready in the event that the problem recurs and is determined to be a hardware fault necessitating equipment replacement. We apologise for any inconvenience caused by this unforeseen event.