Lakeside Software experienced a minor incident on June 28, 2023 affecting SysTrack API/UI, lasting 2d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 28, 2023, 03:08 PM UTC
We are currently investigating an issue where some customers are seeing outdated dated data in the cloud tenant UI.
- identified Jun 28, 2023, 03:21 PM UTC
We have identified the problem and have engaged our cloud provider to support resolving the underlaying issue. The issue only affects a limited portion of the product involved with loading historical data.
- identified Jun 28, 2023, 04:11 PM UTC
We are continuing to work with our cloud partner to resolve the underlaying issue. We will continue to post updates as we get them.
- identified Jun 28, 2023, 05:45 PM UTC
We are continuing to work with our cloud partner to resolve the underlaying issue. We will continue to post updates as we get them.
- identified Jun 28, 2023, 06:57 PM UTC
We are continuing to work with our cloud partner to resolve the underlaying issue. We will continue to post updates as we get them.
- identified Jun 28, 2023, 08:03 PM UTC
We are continuing to work with our cloud partner to resolve the underlaying issue. We will continue to post updates as we get them.
- identified Jun 28, 2023, 10:06 PM UTC
In collaboration with our cloud provider, we have identified a specific underlaying cloud component as being the source of the issue. We are continuing to work closely with our cloud provider to resolve the issue. We will post additional updates as they become available.
- identified Jun 28, 2023, 11:46 PM UTC
In collaboration with our cloud provider, we have identified a specific underlaying cloud component as being the source of the issue. We are continuing to work closely with our cloud provider to resolve the issue. We will post additional updates as they become available.
- identified Jun 29, 2023, 12:45 AM UTC
In collaboration with our cloud provider, we have identified a specific underlaying cloud component as being the source of the issue. We are continuing to work closely with our cloud provider to resolve the issue. We will post additional updates as they become available.
- monitoring Jun 29, 2023, 01:27 AM UTC
The root cause has been remediated by our cloud provider. Some limited historical data could experience residual delays as processing resumes. We will continue to post status updates every 6 hrs until all historical data has successfully processed.
- monitoring Jun 29, 2023, 06:35 AM UTC
Processing of historical data has successfully resumed. We are still processing our backlog and will continue to update every 6 hours.
- monitoring Jun 29, 2023, 12:48 PM UTC
Processing of historical data has successfully resumed. We are still processing our backlog and will continue to monitor and provide update every 6 hours.
- monitoring Jun 29, 2023, 06:30 PM UTC
Processing of historical data has successfully resumed. We are still processing our backlog and will continue to monitor and provide update every 6 hours.
- monitoring Jun 30, 2023, 12:36 AM UTC
Processing of historical data has successfully resumed. We are still processing our backlog and will continue to monitor and provide update every 6 hours.
- monitoring Jun 30, 2023, 06:30 AM UTC
Processing of historical data has successfully resumed. We are still processing our backlog and will continue to update every 6 hours.
- monitoring Jun 30, 2023, 01:46 PM UTC
Processing of historical data has successfully resumed. We are still processing our backlog and will continue to update every 6 hours.
- resolved Jun 30, 2023, 06:22 PM UTC
We have identified the root cause, implemented a fix, and all systems have been fully restored. We will continue to closely monitor all services, but if you have any issues, please contact Lakeside Support at [email protected].
- postmortem Jul 11, 2023, 02:16 PM UTC
# What was the issue? SysTrack Cloud UK did not process uploaded agent condense files. # What was the impact? The issue affected a limited portion of the product involved with loading historical data. # What was the root cause? An underlaying manager service from our cloud provider was not operating as expected. **From our cloud provider**: “We found that a files contained malformed metadata as a result of a delete operation where the system did not properly clean up all metadata associated with the file. Even though the files were deleted, some metadata was present that makes it seem to the application that the file existed. But since the file was partially cleaned up after the deletion, any access request from the clients \(ie opens over SMB\) failed and timed-out due to this incomplete metadata state” # What is the Prevention Strategy? 1. Follow up with our cloud provider to ensure that they complete their remediation steps. 2. DevOps team will add additional alerts via the Azure API to notify us of latency at the storage account layer to catch an issue quicker. 3. Migrate, when possible, away from Azure Files to Azure Blob storage.