Lakeside Software Outage History

# What was the issue? NLQ Agent queries responses could not be guaranteed to be processed. # What was the root cause? See Microsoft public RCA located here: [https://azure.status.microsoft/en-us/status/history/](https://azure.status.microsoft/en-us/status/history/) under “Preliminary Post Incident Review \(PIR\) - Azure Front Door - Connectivity issues across multiple regions” # What is the Prevention Strategy? Design a failover back system if Azure Front Door is not available.

# What was the issue? Some customers experienced slowness when accessing the SysTrack CE UI in the America's region. # What was the root cause? An under-layering resource was over utilized causes the slowness in the UI. # What is the Prevention Strategy? 1. Query data from the underlaying resource less often and implement cache where possible. **\(Completed\)** 2. Add additional alerting \(12/1\)

# What was the issue? SIP features were not responding for customers # What was the root cause? From Microsoft: “Between **18:00 UTC on 24 October 2025** and **00:41 UTC on 25 October 2025**, a platform issue caused intermittent authentication failures and 401 unauthorized errors for services leveraging VNET and/or Private Endpoints across multiple regions \(East US 2, Central US, Australia East, West US, and UK South\). This was due to a conflict in managed identity assignment stemming from an internal configuration change.” # What is the Prevention Strategy? Better error handling within the application

# What was the issue? Access to the SysTrack CE America’s cloud was slow due to sensor processing. # What was the root cause? Preliminary root cause, which is still being verified, was an Azure networking issue causing the utilization of our caching service to respond slowly to requests. This slowness caused cascading slowness to other areas of the platform. # What is the Prevention Strategy? 1. Running a large cache until other steps have been taken. **\(Completed\)** 2. Migrations to a different networking path to access the cache **\(Sept 2025\)** 3. Updates to how the system gets the cache information for recovery quicker **\(Oct 2025\)** 4. Updating the Sensor agent upload mechanism to implement resends or retries. **\(TBD\)**

# What was the issue? Portions of SIP were non-operational and returned error code 500. # What was the root cause? Between 13:30 UTC and 17:45 UTC on 09 May 2025, Microsoft Azure OpenAI Service stopped responding to requests. # What is the Prevention Strategy? 1. Microsoft has posted its RCA and has provided multiple remediation steps for when the Azure OpenAI Service is not responding. \(Tracking Code: LSLX-3WG\) 2. Lakeside is adding additional internal alerts for when Microsoft Azure OpenAI Service is not responding. \(7/1\)

# What was the issue? Less than 4% of our customers had login issues when using SAML SSO following the 11.2 upgrade. # What was the root cause? During our standard SDLC, we updated a software package to its latest version. This update introduced stricter controls on the IDP Entity Setting, preventing login if customers didn't adhere to the recommended Entity setting. # What is the Prevention Strategy? Lakeside 1. worked directly with customers to update their SAML SSO Settings \(Completed\) 2. will preform additional testing for library updates \(On Going\)

During a routine backend-scaling operation, the Microsoft cluster hosting the databases lacked sufficient resources to scale to the desired size. This caused a brief database outage while resources were reallocated. A case has been opened with Microsoft to prevent future occurrences.

# What was the issue? End users had intermittent access issues with the product. # What was the root cause? A backend scaling issues cause some connections to respond back. # What is the Prevention Strategy? 1. Add additional compute \(Completed on day of incident\) 2. Split backend components to allow them to scale independently \(Expect ETA 12/15\)