Lakeside Software experienced a major incident on October 31, 2023 affecting SysTrack API/UI, lasting 6h 43m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 31, 2023, 04:07 PM UTC
We are currently experiencing a performance issue in this region causing the site to load slower than normal. We are still investigating and will provide an update within an hour.
- investigating Oct 31, 2023, 04:08 PM UTC
We are still investigating and will provide an update within an hour.
- identified Oct 31, 2023, 05:47 PM UTC
We are still investigating and will provide an update.
- monitoring Oct 31, 2023, 06:25 PM UTC
We have implemented a fix and are actively monitoring the service.
- resolved Oct 31, 2023, 08:49 PM UTC
This incident has been resolved.
- postmortem Nov 08, 2023, 08:12 PM UTC
# What was the issue? Some clients experienced slowness or 500 errors when accessing the SysTrack Website or APIs. # What was the root cause? One of our databases that stores several admin and access controls functions had a few queries that got into a locked, and blocked state which caused some issues when loading the website or APIs for certain queries. # What is the Prevention Strategy? 1. **Short Term** 1. Understanding which queries they are and optimizing them so they do not lock and block other queries. 2. **Long Term** 1. Rearchitecting the single database into different components so this not becoming a single point of failure