Lakeside Software experienced a major incident on January 22, 2025 affecting SysTrack API/UI and SysTrack Endpoint Connections, lasting 2h 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 22, 2025, 09:45 AM UTC
We are currently investigating reported issues with SysTrack Cloud availability in the below regions. Some users may not be able to use the SysTrack cloud UI, make API calls, or use third-party integrations that utilize the API. We're actively working on identifying the root cause. We apologize for any inconvenience and will provide an update once more details become available.
- identified Jan 22, 2025, 10:14 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jan 22, 2025, 10:15 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 22, 2025, 12:22 PM UTC
This incident has been resolved.
- postmortem Feb 04, 2025, 04:37 PM UTC
# What was the issue? A limited set of end users had intermittent access issues with the product. # What was the root cause? The Azure load balancer service did not scale properly. # What is the Prevention Strategy? 1. Add additional compute \(Completed on day of incident\) 2. Adjustment of alarms for smaller deployments to ensure enough compute is available when an zonal outage occurs. \(Expected 2/28\)