SchemeServe incident

Issue when creating MTA and Renewal case records

Minor Resolved View vendor source →

SchemeServe experienced a minor incident on October 26, 2023 affecting 🎩 SchemeServe, lasting 8m. The incident has been resolved; the full update timeline is below.

Started
Oct 26, 2023, 08:17 AM UTC
Resolved
Oct 26, 2023, 08:26 AM UTC
Duration
8m
Detected by Pingoru
Oct 26, 2023, 08:17 AM UTC

Affected components

🎩 SchemeServe

Update timeline

  1. investigating Oct 26, 2023, 08:26 AM UTC

    We are currently investigating an issue affecting the ability to create new quotations within SchemeServe.

  2. resolved Oct 26, 2023, 08:36 AM UTC

    The issue has been identified and remediation applied, a full post-mortem will be provided soon.

  3. postmortem Oct 27, 2023, 02:30 PM UTC

    In order to future-proof SchemeServe we are proactively increasing the datatype of an identity column in the database. An additional column was required to be added to the table to begin the migration from INT to BIGINT over the coming months. This change was tested successfully within development environments however due to a mirroring of the table for archive purposes onto a second table the schemas were out of sync causing an error when attempting to move values from one to the other. As soon as this was identified the additional column was removed stopping the error whilst additional testing around the specific circumstances that caused the error could be checked. Once the update to both tables had been verified the additional column was then updated again. This change only affected MTAs and renewals and not first premium caserecords as originally believed and so the title of this incident has been updated to reflect this. All cases affected would have created a referral stating the error that can now be fixed by clearing the referral and re-running the caserecord. The change was made after working hours on Wednesday 25th at approximately 17:36 First reported on Thursday 26th at 09:15 Identified and rolled back at 09:26