Compass NZ experienced a major incident on March 24, 2023 affecting Email (Exchange), lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 24, 2023, 05:51 AM UTC
We have identified an issue with the Compass network which is affecting Managed-Services and Voice customers, our team is currently working to resolve the issue.
- identified Mar 24, 2023, 07:01 AM UTC
Teams are onsite working to restore managed services which include: * Terminal Services * Citrix and thin client access * Outlook Exchange * Voice services * Compass website and some internal systems No ETR has been given at this time
- identified Mar 24, 2023, 08:13 AM UTC
Affected hardware has been restarted with PetaSAN being worked on with management console. All managed services will remain unavailable until host management is restored which is current priority. Those customers who lost Internet connectivity before or during the outage should now be able to reconnect. No ETR is yet available
- identified Mar 24, 2023, 09:18 AM UTC
Unfortunately, some systems remained unresponsive and a cold restart has been undertaken. Teams will continue to remain onsite until full restoration has been achieved however providing an ETR at this time is not possible due to the problem's changing landscape.
- monitoring Mar 24, 2023, 06:13 PM UTC
As at approximately 0320 hrs, managed services have been restored to Compass customers: * Terminal Services * Citrix and thin client access * Outlook Exchange * Voice services * Compass website and some internal systems Staff continue to monitor the situation. If you are still experiencing issues with any Compass managed services, please reach out through your designated customer care channels.
- monitoring Mar 24, 2023, 07:28 PM UTC
Confirming all Compass Internal and Managed Customer services have been restored with the exception of Citrix7.2 webpage which is being addressed at present. Staff to continue to monitor systems for any alarms.
- identified Mar 24, 2023, 09:37 PM UTC
Citrix is now fully operational. Compass Customer Care 0800 is currently unable to answer Customer calls. Please head to www.compass.net.nz to chat live or email [email protected]. technicians currently investigating.
- monitoring Mar 25, 2023, 01:57 AM UTC
Systems fully restored with backup call centre ability in operation. Customers are asked to only call for priority support as account and billing inquiries will be deferred until Monday. Full service on our 0800 numbers expected to be restored before Monday morning.
- resolved Mar 25, 2023, 07:28 AM UTC
Continuous monitoring of the previously affected Servers & PetaSAN have revealed no anomalous results since restoration early this morning. Staff are working on future mitigation and contingency plans. Thank you for your patience during this time period.