Lakeside Software experienced a critical incident on August 30, 2023 affecting SysTrack API/UI and SysTrack Endpoint Connections, lasting 13h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 30, 2023, 12:34 PM UTC
We are currently experiencing an outage on our Australia Instance of SysTrack (https://cloud-au.lakesidesoftware.com) due to an ongoing Microsoft outage (https://azure.status.microsoft/en-us/status). We will provide updates as soon as they are available from our cloud provider.
- identified Aug 30, 2023, 01:38 PM UTC
We are actively monitoring the Microsoft Azure Australia East Region outage. We will provide updates as soon as we are receive them.
- identified Aug 30, 2023, 02:41 PM UTC
We are actively monitoring the Microsoft Azure Australia East Region outage. We will provide updates as soon as we are receive them.
- identified Aug 30, 2023, 03:47 PM UTC
We are actively monitoring the Microsoft Azure Australia East Region outage. We will provide updates as soon as we are receive them.
- identified Aug 30, 2023, 06:22 PM UTC
We are seeing some recovery of services as Microsoft Azure fixes the underlying problems. The SysTrack product is mostly operation but we are seeing some intermittent issues. Connections to agents are not operational yet. We will continue to monitor the situation.
- monitoring Aug 30, 2023, 07:50 PM UTC
Even though Microsoft has not yet cleared the underlying issue, we are seeing all services are recovered. We are currently testing the SysTrack product to ensure complete functionality. We will continue to monitor the situation.
- monitoring Aug 30, 2023, 10:59 PM UTC
Even though Microsoft has not yet completely cleared the underlying issue (99% completion of restoration of services https://azure.status.microsoft/en-us/status), we have seen all of our services completely recovered. We have completed our testing of the Australia SysTrack product successfully. Until Microsoft communicates that Azure is back to normal, we will continue to monitor the situation.
- resolved Aug 31, 2023, 02:40 AM UTC
Even though Microsoft has not yet completely cleared the underlying issue (99% completion of restoration of services https://azure.status.microsoft/en-us/status), we have seen all of our services completely recovered. We have completed our testing of the Australia SysTrack product successfully. We will continue to monitor the situation, but are going to mark the incident as resolved as it seems the root cause has been solved by Microsoft.
- postmortem Nov 08, 2023, 08:11 PM UTC
# What was the issue? Our Lakeside Software SysTrack Cloud in Australia cloud was not available. # What was the impact? All customers that use the SysTrack Cloud in Australia were not able to access the website or upload data via the agents. No data on the cloud or the agents were lost due to our application design. # What was the root cause? Microsoft Azure had a cooling issue [\(Tracking ID VVTQ-J98\) ](https://status.azure.com/en-us/status/history/)between 11:34 UTC and **1**9:50 UTC on 30 August 2023. # What is the Prevention Strategy? We are still working with our cloud provider \(Azure\) to understand why our multi zonal configuration was effect with a single zonal outage. We will update this RCA when this is identified with our prevention strategy.