Core Practice Outage History

Summary: Between 07:00 AEST and 13:00 AEST, a number of users were unexpectedly logged out when attempting to generate invoices. The issue was caused by a regression in the most recent application release. A hotfix was deployed between 12:30pm and 1:00pm AEST and testing confirmed users are no longer being logged out. We will continue to monitor closely. Investigation: Telemetry indicated a consistent pattern of authorisation failures on an endpoint called during the invoice generation workflow. Further review identified that recent changes were causing some users to be logged out when this response was received, rather than handling it gracefully. Next Steps: The hotfix has corrected the client-side handling of these responses. Some users may need to refresh their browser to receive the updated application files. A problem ticket has been raised to review our pre-release testing process, specifically to ensure least-privileged user scenarios are covered in regression testing for permission-sensitive workflows. We apologise for the inconvenience caused and thank affected users for their patience.

Status: Resolved Summary: Between 16:20 and 16:45, we experienced elevated CPU utilisation across multiple instances, which caused intermittent slowness for some users. The affected servers have since stabilised and were progressively returned to the pool, with all instances restored by approximately 17:20. Investigation: Initial monitoring through DataDog did not show a common request pattern across the affected servers. Further review of instance logs indicates a consistent sequence of events: spikes in network traffic → RAM saturation → CPU spikes. Next Steps: A detailed log analysis is currently underway to determine the root cause. A problem ticket has been raised to track the investigation and any corrective actions. We apologise for any inconvenience caused and will continue to monitor system performance closely.

Between 10:55am and 12:05pm 25/02/2026, we saw resource spikes on some of our front end instances. The issue has been resolved, additional servers were provisioned to stabilise performance and server load has returned to normal levels. Response times are back to expected benchmarks. Root cause investigation is ongoing. A post-incident report will be published once completed.

Twilio has confirmed the issue impacting inbound SMS messages for a subset of Australian phone numbers has been resolved. We are no longer seeing delivery issues and inbound SMS messages are functioning as expected. Outbound SMS messages were not impacted during this incident. We will continue to monitor, but no further action is required at this time.

Resolution Summary We sincerely apologise for the inconvenience caused. This incident resulted from a sequence of compounding events that amplified the impact and extended the resolution time longer than expected. Our emergency response processes were initiated, however, due to external factors, they did not function as effectively as intended. For full transparency, please see the details below. Overview Between 8:00am and 9:50am AEST on 2 February 2026, Core Practice experienced a performance degradation due to a failure in the VM Autoscale automation capability. As a result, newly created virtual machines were unable to successfully install startup configurations, causing them to report an unhealthy state. This led to reduced VM capacity across the platform. 8:34am AEST, We became aware of the issue following customer reports of system slowness. 8:42am AEST, Our engineering team initiated emergency procedures to manually override VM configurations and restore individual virtual machines to a healthy state. In parallel, a separate team began provisioning a new VM pool to restore capacity. 9:15am AEST, We identified ongoing service issues reported by our service provider, Microsoft Azure. Relevant incidents published on the Azure Status page included: https://azure.status.microsoft/en-us/status 19:46 UTC, 2 February 2026 – Incorrect notifications during VM management operations 00:02 UTC, 3 February 2026 – Issues related to storage accounts and host extension packages Issue Amplification The incident was further compounded by Azure reporting incorrect VM health and provisioning states. While engineers were manually overriding configuration files, Azure Monitoring continued to display affected VMs as unhealthy. Under normal circumstances, these VMs should have reported a healthy status. This misleading information diverted engineering efforts toward investigating additional, non-existent issues. 9:40am AEST, We confirmed that the manually overridden VMs were operating correctly and that the primary remaining issue was inaccurate health and provisioning status reporting within Azure Monitoring. Engineers promptly updated the configurations on the remaining VMs to restore full system functionality. 9:50am AEST All systems were fully restored and operating normally. Root Cause The root cause of this incident was an ongoing service issue within Microsoft Azure that affected storage accounts used by VM custom extension packages. These storage accounts are required during VM provisioning to startup configuration routine. This issue was further exacerbated by Azure reporting incorrect VM health and provisioning states, which delayed identification of the true operational status of affected virtual machines. Next Steps We sincerely apologise for the impact this incident had on affected customers. We are continuing to work closely with Microsoft Azure to: Improve the reliability and accuracy of health and provisioning state reporting. Improve our emergency response notification and procedures. Investigate architectural improvements to increase system resiliency and automation to reduce resolution time for future incidents.