Lakeside Software incident
SysTrack Cloud – Canada – Reports of Delayed Performance
Lakeside Software experienced a major incident on March 22, 2024 affecting SysTrack API/UI and SysTrack Endpoint Connections, lasting 14h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 22, 2024, 05:17 AM UTC
We are continuing to investigate this issue.
- investigating Mar 22, 2024, 05:19 AM UTC
We are continuing to investigate this issue.
- identified Mar 22, 2024, 06:51 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 08:00 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 09:00 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 10:00 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 11:02 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 12:00 PM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 01:00 PM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 02:07 PM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 02:59 PM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 22, 2024, 04:05 PM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- monitoring Mar 22, 2024, 05:14 PM UTC
Microsoft has remediated the problem. All services are now operational. We will continue to monitor the situation and provide updates approximately every 4 hours
- resolved Mar 22, 2024, 07:43 PM UTC
This incident has been resolved. Azure has identifying the root cause and mitigated the issue. We apologize for the inconvenience, If you're still experiencing performance problems, please reach out to [email protected].
- postmortem Apr 04, 2024, 01:58 AM UTC
# What was the issue? All customers experienced intermittent problems loading the SysTrack CE for a period of time. Agents as well as users of the SysTrack web product were unable to connect for periods of time: # What was the root cause? This is a preliminary RCA that is subject to change once we get the final RCA from Microsoft. Thus far, we have determined the root cause to be with the Microsoft Azure’s Application Gateway. This managed service, fully supported by Microsoft, appears to have had a backend update which was deployed by Microsoft to the different SysTrack regions. This updated version does not appear to handle our unique workload on the Application Gateway. The managed service got overloaded and caused it to go into an _unavailable_ state thus not accepting any inbound traffic. # What is the Prevention Strategy? We are still working with our cloud provider \(Azure\) to get a full RCA before we prepare an identification and prevention strategy.