Lakeside Software incident
SysTrack Cloud - Germany – Reports of Delayed Performance
Lakeside Software experienced a minor incident on March 26, 2024 affecting SysTrack API/UI and SysTrack Endpoint Connections, lasting 3h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 26, 2024, 09:08 AM UTC
Some slowness when using the SysTrack Cloud UI or while making API calls. We're actively working on identifying the root cause. We apologize for any inconvenience and will provide an update once more details become available
- identified Mar 26, 2024, 09:08 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 26, 2024, 09:59 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 26, 2024, 10:59 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- identified Mar 26, 2024, 11:59 AM UTC
In cooperation with Microsoft, we have identified the root cause of the problem. The issue resulted from a recent change to Microsoft's environment. We are working closely with Microsoft to restore all services and mitigate the risk of any reoccurrence.
- resolved Mar 26, 2024, 12:14 PM UTC
In coordination with Microsoft, we believe this issue is resolved.
- postmortem Apr 04, 2024, 01:57 AM UTC
# What was the issue? All customers experienced intermittent problems loading the SysTrack CE for a period of time. Agents as well as users of the SysTrack web product were unable to connect for periods of time: # What was the root cause? This is a preliminary RCA that is subject to change once we get the final RCA from Microsoft. Thus far, we have determined the root cause to be with the Microsoft Azure’s Application Gateway. This managed service, fully supported by Microsoft, appears to have had a backend update which was deployed by Microsoft to the different SysTrack regions. This updated version does not appear to handle our unique workload on the Application Gateway. The managed service got overloaded and caused it to go into an _unavailable_ state thus not accepting any inbound traffic. # What is the Prevention Strategy? We are still working with our cloud provider \(Azure\) to get a full RCA before we prepare an identification and prevention strategy.