Reports of slow speeds

Update timeline

1. investigating Mar 06, 2025, 09:19 AM UTC

   We are currently investigating reports of slow speeds affecting some users. Our team is working to resolve this as quickly as possible. Thank you for your patience.

2. monitoring Mar 06, 2025, 11:36 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 06, 2025, 03:03 PM UTC

   This incident has been resolved.

4. postmortem Mar 07, 2025, 07:15 AM UTC

   **Root Cause** On Wednesday 5th, between 10:00 PM and 2:00 AM, we carried out an upgrade of our primary database. Following this upgrade, a subset of complex queries began executing with less-than-optimal query plans, resulting in a significant slowdown of over 100 times compared to previous performance. This issue was not identified during extensive pre-upgrade testing in a separate environment, where query performance remained stable over several months. However, in the live environment, the database optimiser selected different execution plans, leading to the unexpected performance degradation. ‌ **Impact** The issue affected all customers when performing case searches, significantly increasing query execution times. ‌ **Detection** We became aware of the issue on Thursday, 6th March, at 6:30 AM, when reports of slow performance were received. The excessive query execution times caused a knock-on effect across all usage of SchemeServe, as the additional load exceeded 100 times the normal peak load, further impacting overall system performance. ‌ **Resolution** Following our investigation, we determined that reverting the affected queries to SQL 2019 compatibility mode would restore their performance. This change was implemented immediately, and normal service levels were fully restored. ‌ We sincerely regret any disruption caused and appreciate our customers’ patience while we worked to resolve the issue.