Compass NZ experienced a major incident on March 1, 2023 affecting Fibre Broadband and Copper Broadband (xDSL), lasting 1d 12h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 01, 2023, 07:27 PM UTC
Loss of power to a partner's data centre has meant the authentication services for Compass retail customers has gone offline. Most customers will not have internet access. Technicians are onsite restoring power. Following restoration, services will be restarted and begin to come online throughout the morning. We thank you for your patience this morning and respectfully ask that if you do not have another reason to contact Compass Customer Care, that you stand by for updates here.
- investigating Mar 01, 2023, 07:29 PM UTC
Servers have been restarted and coming online. Some customers may have regained access however this may be lost again throughout the morning as work continues.
- monitoring Mar 01, 2023, 07:30 PM UTC
Servers have been restarted and coming online. Some customers may have regained access however this may be lost again throughout the morning as work continues.
- monitoring Mar 01, 2023, 07:53 PM UTC
Compass business support systems are also affected and staff are working to restore customer-facing elements. This includes phone and email which are not currently operational. Updates to follow as they are received.
- monitoring Mar 01, 2023, 08:34 PM UTC
Internet connections are being restored for all Compass customers. This will also be restoring phone services provided over fibre and xDSL connections. If you are not yet online, restarting your router or modem may resolve the connection. With many customers trying to reconnect at once, you may need to restart more than once to be successful. Some business customers with SIP trunk and cloud PBX services are still offline and being worked on presently. Compass Care 0800 numbers are not yet restored and may report being disconnected or unavailable. A reminder again to customer that we appreciate your patience while we restore services to all of our customers and ask that you only attempt to contact us with inquiries unrelated to the present outage.
- monitoring Mar 01, 2023, 11:47 PM UTC
Fibre, xDSL, home phone and most business voice products have been restored and are being monitored. Please expect minor interruptions throughout the afternoon as the network stabilizes. All Compass customers are presently unable to make or receive calls to/from the Spark voice network including both mobile and landline numbers. Our teams are investigating and will be planning a resolution once root cause is identified. Compass internal customer support systems are mostly restored. We remind customers to restart their routers to restore services and avoid calling and emailing for non-urgent inquires. Thank you for your ongoing patience while we restore the network and your services.
- monitoring Mar 02, 2023, 01:47 AM UTC
The previously reported, major outage has been resolved but we continue to monitor the network to ensure stability across all platforms. Internet: All user should now be online. If you have not reconnect yet, please restart your router or modem. Please do not reset your modem or router. For those who have, please find your router configuration guide at https://compass.net.nz/help-support/user-guides and scroll to Router Settings. Business and Managed Services: All service layers have been restored but some may be subject to latency as demand levels return to normal. Voice: All voice services including Cloud PBX, copper and fibre landline have been restored however, calls to and from all Spark numbers are currently unavailable. This is being worked on and our primary priority at present. Other: Webmail and Compass email services have been restored however email delivery will be delayed as all the messages are processed.
- monitoring Mar 02, 2023, 03:33 AM UTC
The previous major outage affecting the Compass network has been resolved. Internet, voice, cloud services as well as the earlier Spark calling problems have been addressed and the restoration teams responsible will continue to monitor closely over the next 24 hours. Customers are reminded that restarting equipment that has failed to reconnect, will restore your connection. Please do not reset or factory-reset equipment. If you have done so, please visit https://compass.net.nz/help-support/user-guides for information on reconfiguring your equipment. We are grateful for your patience during this outage.
- monitoring Mar 02, 2023, 07:41 PM UTC
Good morning. Teams continue to monitor Compass network elements today following yesterday's outage. Retail residential and business customers should all now be online without issue. Please contact Compass Customer Care if issues persist for you. 0800 840 640 or [email protected] Managed Services: HyperV is still showing signs of instability resulting in some virtual machines becoming unavailable. Exchange and Outlook mail and other managed services are affected. The team responsible is working hard to bring stability back to these services but some customers will experience degradation or loss of service for brief periods as vm-compute nodes are restarted. Updates will be posted as they are made available by the teams.
- monitoring Mar 03, 2023, 12:14 AM UTC
Managed Services: Our teams have completed rebooting HyperV vm-compute nodes one at a time. This has resulted in stabilsing the platform. Outlook / Exchange mail service has been restored. Citrix XenApp virtual machines still have some remaining issues and need to be manually re-started. ETA for repair is 1 hour and an update will follow before 1430
- resolved Mar 03, 2023, 02:03 AM UTC
Apologies for the delayed update due before 1430. Managed Service: All but a small number of customers are restored with isolated issues being attended to. Our restoration teams remain in direct contact as necessary with those with outlying symptoms. Customer impacts are now low and being treated on a case by case basis. If you are not currently experiencing degraded or no service on a Compass product, please contact Compass on 0800 840 640. No further updates follow.