Lakeside Software incident

SysTrack Cloud – Reports of Delayed Performance

Major Resolved View vendor source →

Lakeside Software experienced a major incident on January 29, 2024 affecting SysTrack API/UI and SysTrack API/UI and 1 more component, lasting 7h 26m. The incident has been resolved; the full update timeline is below.

Started
Jan 29, 2024, 12:45 PM UTC
Resolved
Jan 29, 2024, 08:12 PM UTC
Duration
7h 26m
Detected by Pingoru
Jan 29, 2024, 12:45 PM UTC

Affected components

SysTrack API/UISysTrack API/UISysTrack API/UI

Update timeline

  1. investigating Jan 29, 2024, 12:45 PM UTC

    Some slowness when using the SysTrack Cloud UI or while making API calls. We're actively working on identifying the root cause. We apologize for any inconvenience and will provide an update once more details become available

  2. investigating Jan 29, 2024, 01:01 PM UTC

    We are continuing to investigate this issue.

  3. investigating Jan 29, 2024, 01:02 PM UTC

    We are continuing to investigate this issue.

  4. identified Jan 29, 2024, 01:10 PM UTC

    The issue has been identified and a fix is being implemented.

  5. monitoring Jan 29, 2024, 01:38 PM UTC

    A fix has been implemented and we are monitoring the results.

  6. investigating Jan 29, 2024, 02:19 PM UTC

    We are currently investigating some issues loading specific pages on the website. We're actively working on identifying the root cause. We apologize for any inconvenience and will provide an update once more details become available.

  7. monitoring Jan 29, 2024, 03:03 PM UTC

    A fix has been implemented and we are monitoring the results.

  8. resolved Jan 29, 2024, 08:12 PM UTC

    This incident has been resolved.

  9. postmortem Feb 05, 2024, 02:57 PM UTC

    # What was the issue? Some clients experienced slowness, non-responsiveness or HTTP 50X errors when accessing the SysTrack Website or APIs. # What was the root cause? After the 10.11 release, a bug was introduced that overloaded the SQL database causing some calls to the database to be delayed or error out. # What was the resolution? 1. Scaled the database in the short term so it was not overloaded. 2. Fixed the newly introduced bug, tested, and patched production. # What is the Prevention Strategy? 1. Perform additional load testing prior to releases to validate no additional calls are made in an unexpected manner.