SchemeServe experienced a notice incident on March 3, 2025, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Mar 03, 2025, 05:29 PM UTC
On Sunday, March 2nd, between 08:33 and 13:48, SchemeServe experienced an issue that prevented new case records from completing fully, meaning they could not be placed on cover.
- postmortem Mar 03, 2025, 05:30 PM UTC
#### **Root Cause** The issue was caused by a new database backup and replication process that was initiated on Saturday, March 1st, at 14:32. While the process initially appeared to be functioning as expected, it was not until 08:33 on Sunday that the issue manifested, preventing quote submissions. When replication was enabled, each column value had a maximum size limit of 65,536 bytes. The `PolicyRecordDebugLogs` exceeded this limit, leading to errors that blocked new case records from completing. Despite thorough testing in multiple test environments prior to deployment, a misconfiguration in the live environment led to the issue. Unfortunately, this was due to human error in applying the necessary settings correctly during implementation. #### **Resolution** Once the issue was identified, corrective action was taken, and normal service was restored by 13:48 on Sunday. #### **Preventative Measures** To prevent similar issues in the future, we have implemented additional safeguards, including: * Enhanced deployment checks to ensure live configurations match tested environments. * Additional validation steps in our change management process. * Improved monitoring and alerting to detect similar issues earlier and prevent service disruptions. We sincerely regret the disruption caused and take full responsibility for the error. We are committed to improving our processes to ensure that such incidents do not occur again.