FireNet incident
Ip Transit and Internet degraded performance in NSW
FireNet experienced a major incident on December 6, 2020 affecting Co-location and IP Transit, lasting 12h 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 07, 2020, 12:33 AM UTC
We have identified that a number of IP transit and Internet services in NSW are experiencing degraded performance. Our carrier is currently investigating the issue. Further updates to follow. Please contact the FireNet NOC should you have any questions, comments, or concerns. Email: [email protected] Phone one of the following numbers: Australia: 1300 636 636 International customers please call +61 2 9397 7050
- investigating Dec 07, 2020, 12:42 AM UTC
We are continuing to investigate this issue.
- monitoring Dec 07, 2020, 04:34 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Dec 07, 2020, 11:23 AM UTC
Impact Summary At approximately 09:49 Monday 7th December 2020 AEDST, for up to 120 minutes, FireNet observed disruptions to a number of transit and peering connections within our New South Wales edge devices causing an internet outage for a majority of services. Technical Brief FireNet engineers received alerts from internal systems for our edge devices notifying high cpu and memory usage. Upon investigation we noticed all BGP sessions in a hung state where they held current routes but were not updating. The devices were accepting commands but not applying. We were able to re-route traffic to other nodes and have now identified the issue that affected the nodes. Moving Forward FireNet engineers have sent through logs of the affected devices to the vendor to identify the root cause. Once the logs have been analysed, we will work with the vendor to test and deploy changes to these nodes. FireNet will notify of any maintenance in due course.