LoanMAPS incident

Degraded Performance: Database Server Under High Load

Minor Resolved View vendor source →

LoanMAPS experienced a minor incident on October 31, 2019, lasting 3h 52m. The incident has been resolved; the full update timeline is below.

Started
Oct 31, 2019, 06:16 PM UTC
Resolved
Oct 31, 2019, 10:09 PM UTC
Duration
3h 52m
Detected by Pingoru
Oct 31, 2019, 06:16 PM UTC

Update timeline

  1. investigating Oct 31, 2019, 06:16 PM UTC

    We are aware of an issue causing our database server to be under high load. This is impacting the general performance of the system. This issue is being actively investigated.

  2. identified Oct 31, 2019, 06:18 PM UTC

    We identified an issue with one of our database queries that loads the "Not Assigned" section of the "Pre-Close" tab on the dashboard. We are working to improve the performance of this query now.

  3. identified Oct 31, 2019, 06:38 PM UTC

    Clarification to the previous update: The query in question is causing overall high load on the database server, which is impacting other areas of the system.

  4. identified Oct 31, 2019, 07:51 PM UTC

    We are currently working on testing a possible fix for this issue and will have more information to share shortly.

  5. identified Oct 31, 2019, 08:48 PM UTC

    We have finished updating the query and are deploying it into the production environment now.

  6. resolved Oct 31, 2019, 10:09 PM UTC

    This issue was fixed in production at 3:00 PM. Monitoring for the past hour shows that it has been resolved.