Lakeside Software incident

Lakeside Software Cloud Services - Degraded Performance - Insiders

Lakeside Software experienced a minor incident on July 4, 2023 affecting SysTrack API/UI, lasting 28m. The incident has been resolved; the full update timeline is below.

Started: Jul 04, 2023, 03:45 PM UTC
Resolved: Jul 04, 2023, 04:14 PM UTC
Duration: 28m
Detected by Pingoru: Jul 04, 2023, 03:45 PM UTC

Affected components

SysTrack API/UI

Update timeline

investigating Jul 04, 2023, 03:45 PM UTC

Lakeside Software is currently experiencing degraded performance with one or more services impacting the below regions. During this period, users might experience some slowness when using the SysTrack cloud UI or while making API calls. We're actively working on identifying the root cause and will resolve the problem as soon as possible. We apologize for the inconvenience and will provide an update as soon as all services are operational.
resolved Jul 04, 2023, 04:14 PM UTC

Issue has been resolved. We have identified the root cause, implemented a fix, and all systems have been fully restored. We will continue to closely monitor all services, If you're still experiencing performance problems, please reach out to [email protected].
postmortem Jul 13, 2023, 05:06 PM UTC

# What was the issue? Customers experienced slow performance when accessing the [insiders.lakesidesoftware.com](http://insiders.lakesidesoftware.com/) website. # What was the root cause? A cloud infrastructure alerting script was deployed that used more than expected resources. # What is the Prevention Strategy? * Review the process where internal backend automations scripts are deployed. * Review SQL alarming metrics.