SchemeServe Outage History

Started Aug 05, 2026, 11:42 AM UTC· resolved Aug 05, 2026, 11:55 AM UTC

1. investigating Aug 05, 2026, 11:42 AM UTC

   We are investigating an issue causing reports to not generate. We are working on a resolution with urgency.

2. resolved Aug 05, 2026, 11:55 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 01, 2026, 02:41 PM UTC· resolved Jul 01, 2026, 02:47 PM UTC

1. investigating Jul 01, 2026, 02:41 PM UTC

   We are investigating an issue where reports are filing to generate. We are looking into this with urgency to get a resolution as soon as possible.

2. resolved Jul 01, 2026, 02:47 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 18, 2026, 12:27 PM UTC· resolved Jun 18, 2026, 12:34 PM UTC

1. investigating Jun 18, 2026, 12:27 PM UTC

   We are investigating an issue affecting the deliverability of email multi-factor-authentication and deliverability of reports. We are investigating this at the highest urgency.

2. resolved Jun 18, 2026, 12:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 21, 2026, 10:53 AM UTC· resolved Apr 21, 2026, 11:56 AM UTC

1. investigating Apr 21, 2026, 10:53 AM UTC

   We are investigating an issue where certain reports are failing to generate correctly. We are looking into this at our highest priority to resolve this as soon as possible.

2. resolved Apr 21, 2026, 11:56 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 31, 2026, 02:10 PM UTC· resolved Mar 31, 2026, 02:48 PM UTC

1. investigating Mar 31, 2026, 02:10 PM UTC

   Some of our additional modules are experiencing issues, and may be inaccessible. We are investigating this as a highest priority.

2. resolved Mar 31, 2026, 02:48 PM UTC

   This incident has been resolved.

3. postmortem Apr 01, 2026, 08:48 AM UTC

   ## Incident Report: Partial Frontend Outage on March 31, 2026 ## Issue The automatic renewal of an internal SSL certificate failed, causing our feature flag service to go down. As a result, newer interfaces—QSE and Digit—did not load correctly when the frontend attempted to retrieve feature flags from the service. ## Summary On March 31, 2026, a subset of users experienced issues loading the new frontends \(QSE and Digit\). When the frontend requested an update to feature flags, only the frontend was affected—no other services or processes were impacted. We know how disruptive this was and sincerely regret the inconvenience. ## Timeline * ~12:16: Internal SSL certificate failed to renew automatically. * ~15:00: First reported instance of QSE failing to load. * 15:11: Root cause identified. * 15:46: Service restored. ## Impact * Affected: New frontend screens—QSE, Digit, Translations, Email Logs, Machine Keys. * Duration: Approximately 3.5 hours. ## Root Cause Analysis The internal SSL certificate between Cloudflare and our feature flag service failed to automatically renew as expected. This caused the service to become unreachable, resulting in the frontend either displaying a “white screen of death” or reverting to older UI versions when making network requests to the feature flag API service. ## Resolution Steps * Removed the automated renewal process. * Created a long-lived SSL certificate for this service. ## Preventative Actions * Implemented longer-lived SSL certificates: * Eliminates automated renewal dependence, reducing the risk of unexpected outages. * Enhanced alerting on the uptime state of the feature flag service: * Ensures faster detection and response to potential downtime. ## Apology We sincerely apologize for any disruption this may have caused your business. We are deeply committed to preventing similar incidents in the future. Thank you for your understanding and patience.

Read the full incident report →

Started Jan 08, 2026, 03:26 PM UTC· resolved Jan 09, 2026, 11:26 AM UTC

1. investigating Jan 08, 2026, 03:26 PM UTC

   SchemeServe is currently experiencing intermittent service issues. This is being investigated at the highest priority.

2. monitoring Jan 08, 2026, 05:01 PM UTC

   We are continuing to monitor this.

3. resolved Jan 09, 2026, 11:26 AM UTC

   This incident has been resolved.

4. postmortem Jan 13, 2026, 12:40 PM UTC

   ### Summary On Thursday 8th January 2025 between 14:53 and 17:33, SchemeServe experienced an incident that resulted in intermittent request timeouts for some customers. This occurred when customer requests relied on additional internal service-to-service calls, which were delayed due to unexpected back-pressure within our centralised logging infrastructure. The issue did not affect all customers or all requests, and no data was lost. However, we recognise that any request timeout impacts customer trust, and we sincerely apologise for the disruption. ### Customer Impact During the incident window: * Some customer requests timed out when they depended on an additional internal service call * The impact was intermittent and limited to a subset of workflows * No customer data was lost or corrupted * Platform availability remained intact, but responsiveness was degraded for affected requests * At its peak, this issue affected less than 1% of calls made to SchemeServe We understand the importance of reliability and are committed to ensuring this does not happen again. ### What Happened SchemeServe uses a centralised logging service to collect logs from all internal services. To optimise performance, logs are batched in in-memory queues before being written to the database. On the day of the incident, there was a significant increase in both the volume and size of log messages. Under this usage, the client unintentionally created a new HTTP client per request, leading to a much higher number of concurrent connections than expected. As traffic increased, several compounding factors interacted: 1. In-memory log queues filled faster than they could be drained, increasing memory pressure. 2. Database writes slowed due to connection pool contention, triggering retry logic. 3. Retry delays were longer than intended, causing queue drainage to slow further. 4. Incoming log requests waited for space in the in-memory queue instead of failing fast, holding HTTP connections open. 5. Over time, this exhausted available HTTP sockets. 6. Once sockets were exhausted, other internal service-to-service calls were forced to wait or time out. 7. Customer requests that depended on these internal calls also timed out. In parallel, logs that could not be written immediately were written to storage for later ingestion. This prevented data loss, but when the logging service attempted to ingest these logs in the background, it added additional load at a time when the system was already under pressure. Individually, each mechanism exists to improve resilience. In this case, the combination of increased load, large payload sizes, retry timing, connection handling, and scaling limits created a cascading issue. ### Detection and Mitigation Once the issue was identified: * Log ingestion through the logging service was temporarily disabled * Services automatically fell back to writing logs to storage * This reduced pressure on HTTP connections and allowed internal services to recover * Once traffic returned to normal levels, all stored logs were successfully processed Service performance returned to normal shortly after mitigation was applied. ### Root Cause The root cause was unexpected back-pressure in the logging pipeline caused by a combination of: * Increased log volume and size * Inefficient HTTP client reuse under this usage pattern * Slow retry backoff when database connections were exhausted * In-memory queue behaviour that held connections open rather than failing fast * Limited horizontal scaling under memory pressure These factors interacted in a way that was not observed during prior testing. ### Why This Was Not Detected Earlier Although extensive load testing was performed, this scenario did not surface because it required a specific combination of conditions: * High request throughput * Large individual log payloads * Slower-than-expected retry delays * High concurrency of short-lived HTTP clients * Memory pressure preventing timely scaling * Previous similar ossurances in the production environment had managed successfulyl using designed fallback systems Each condition was tested independently, but the combined effect was not fully represented in test scenarios. ### Actions we are taking to Prevent Recurrence We have implemented the following improvements: * Reduced database retry backoff times to prevent prolonged queue blocking * Increased memory allocation for in-memory log queues * Introduced dedicated HTTP clients for log writes with short, bounded timeouts * Reduced inter-service timeout durations to avoid holding connections unnecessarily * Improved separation of critical and non-critical log ingestion paths We are also adjusting our deployment strategy to roll out changes incrementally and observe system behaviour under real traffic before full rollout. ### Resolution and Recovery The incident was resolved by relieving pressure on the logging service and allowing internal services to recover available connections. Once normal operating conditions returned, all queued and stored logs were processed successfully. ### Closing We apologise for the disruption this incident caused. While the underlying issue occurred within our internal infrastructure, we recognise that the impact was customer-visible and take responsibility for it. The lessons learned from this incident have already led to concrete improvements that strengthen the resilience of our platform. We remain committed to delivering a reliable, performant service and to being transparent when issues occur. If you have questions or would like further detail, please contact the SchemeServe support team.

Read the full incident report →

Started Nov 18, 2025, 11:43 AM UTC· resolved Nov 18, 2025, 03:02 PM UTC

1. investigating Nov 18, 2025, 11:43 AM UTC

   We’re currently experiencing intermittent service disruptions due to Cloudflare issues. Our team is monitoring closely and will update as soon as possible

2. investigating Nov 18, 2025, 11:54 AM UTC

   Cloudflare are currently experiencing global network issues, which are affecting services across the internet. This may cause intermittent disruption. We’re monitoring the situation and will provide updates as Cloudflare restores normal service.

3. monitoring Nov 18, 2025, 02:19 PM UTC

   Unfortunately, Cloudflare continue to experience global network issues, affecting significant parts of the internet and including services such as AWS, PayPal, Visa, BT & Sage. Thank you for your patience - whilst this issue is unfortunately beyond SchemeServe's control, we will keep you informed as we are monitoring the situation closely.

4. resolved Nov 18, 2025, 03:02 PM UTC

   Cloudflare have resolved the global network issue. Services are now fully restored and operating normally. Thank you for your patience.

5. postmortem Nov 19, 2025, 01:31 PM UTC

   On 18 November 2025, a global network incident at Cloudflare caused a service outage across multiple platforms, including SchemeServe. The root cause was a permissions-related configuration change that triggered propagation of an oversized data file, leading to widespread HTTP 5xx errors. We sincerely apologise for the disruption. Normal service has now been fully restored, and Cloudflare has published a full incident report at: [Cloudflare blog](https://blog.cloudflare.com/18-november-2025-outage/?utm_source=chatgpt.com). We are reviewing our continuity and monitoring measures to minimise the risk of similar external-dependency issues in future.

Read the full incident report →

Started Oct 20, 2025, 10:09 AM UTC· resolved Oct 21, 2025, 08:13 AM UTC

1. investigating Oct 20, 2025, 10:09 AM UTC

   Some users may experience issues with email and SMS deliverability. This is due to an outage of AWS, and we are continuing to monitor the situation.

2. resolved Oct 21, 2025, 08:13 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 10, 2025, 03:18 PM UTC· resolved Oct 10, 2025, 03:36 PM UTC

1. investigating Oct 10, 2025, 03:18 PM UTC

   We are currently investigating an issue where some policies are unable to be viewed. We are working on this with the utmost urgency, and will resolve this as soon as possible.

2. resolved Oct 10, 2025, 03:36 PM UTC

   This incident has been resolved.

3. postmortem Oct 10, 2025, 04:30 PM UTC

   On Friday between 16:09 and 16:30 BST, some users briefly experienced issues accessing policy answers data. The cause was a search index update that didn’t complete as expected. This update had recently been moved to a scheduled process to improve predictability and stability. Our team identified and resolved the issue quickly, restoring full access within 20 minutes. The process has since been reviewed and refined to further strengthen resilience. This update was low-risk and had been fully tested in non-production environments with no adverse effects. Services are now operating normally.

Read the full incident report →

Started Mar 31, 2025, 07:09 AM UTC· resolved Mar 31, 2025, 08:37 AM UTC

1. investigating Mar 31, 2025, 07:09 AM UTC

   SchemeServe is currently experiencing issues with multi-factor authentication. the issue is under invetigation.

2. resolved Mar 31, 2025, 08:37 AM UTC

   The incident has been resolved, and MFA is now working correctly again. A post-mortem will be following shortly

3. postmortem Mar 31, 2025, 12:47 PM UTC

   **Root Cause** SchemeServe’s authentication system relies on short-lived access tokens known as JWTs \(JSON Web Tokens\), each with a lifespan of 15 minutes before requiring renewal. Historically, SchemeServe has used **BST \(British Summer Time\)** across its codebase and storage layers. As part of an ongoing effort to align with international standards, newer services have been transitioned to operate in **UTC**. However, the JWT generation process had not yet been updated and was still using BST. As a result, when the UK clocks moved forward at **01:00 on Sunday, 30th March**, newly generated tokens contained timestamps based on BST. These tokens appeared to be issued in the future relative to the UTC-based verification process, placing them immediately outside the valid lifetime window. Consequently, the tokens were rejected and users were unable to authenticate. **Impact** This issue prevented users from authenticating into **SchemeServe**, effectively blocking access to the platform. **Detection** Unfortunately, this event did not trigger any health alerts and as such, the issue was not detected in real-time. We became aware of the problem at **07:50 on Monday, 31st March**, following user reports. Further investigation quickly identified the root cause. The incident led to prolonged authentication failures from **01:00 Sunday** until resolution. **Resolution** The JWT generation process was updated to explicitly use **UTC**, ensuring that newly issued tokens fall within the expected validity window. This change was deployed, and the issue was fully resolved by **09:34 on Monday, 31st March**. We sincerely regret the disruption caused by this incident and appreciate your understanding as we continue to improve the resilience and reliability of the SchemeServe platform.

Read the full incident report →

Started Mar 06, 2025, 09:19 AM UTC· resolved Mar 06, 2025, 03:03 PM UTC

1. investigating Mar 06, 2025, 09:19 AM UTC

   We are currently investigating reports of slow speeds affecting some users. Our team is working to resolve this as quickly as possible. Thank you for your patience.

2. monitoring Mar 06, 2025, 11:36 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 06, 2025, 03:03 PM UTC

   This incident has been resolved.

4. postmortem Mar 07, 2025, 07:15 AM UTC

   **Root Cause** On Wednesday 5th, between 10:00 PM and 2:00 AM, we carried out an upgrade of our primary database. Following this upgrade, a subset of complex queries began executing with less-than-optimal query plans, resulting in a significant slowdown of over 100 times compared to previous performance. This issue was not identified during extensive pre-upgrade testing in a separate environment, where query performance remained stable over several months. However, in the live environment, the database optimiser selected different execution plans, leading to the unexpected performance degradation. ‌ **Impact** The issue affected all customers when performing case searches, significantly increasing query execution times. ‌ **Detection** We became aware of the issue on Thursday, 6th March, at 6:30 AM, when reports of slow performance were received. The excessive query execution times caused a knock-on effect across all usage of SchemeServe, as the additional load exceeded 100 times the normal peak load, further impacting overall system performance. ‌ **Resolution** Following our investigation, we determined that reverting the affected queries to SQL 2019 compatibility mode would restore their performance. This change was implemented immediately, and normal service levels were fully restored. ‌ We sincerely regret any disruption caused and appreciate our customers’ patience while we worked to resolve the issue.

Read the full incident report →

Started Mar 02, 2025, 01:30 AM UTC· resolved Mar 02, 2025, 01:30 AM UTC

1. resolved Mar 03, 2025, 05:29 PM UTC

   On Sunday, March 2nd, between 08:33 and 13:48, SchemeServe experienced an issue that prevented new case records from completing fully, meaning they could not be placed on cover.

2. postmortem Mar 03, 2025, 05:30 PM UTC

   #### **Root Cause** The issue was caused by a new database backup and replication process that was initiated on Saturday, March 1st, at 14:32. While the process initially appeared to be functioning as expected, it was not until 08:33 on Sunday that the issue manifested, preventing quote submissions. When replication was enabled, each column value had a maximum size limit of 65,536 bytes. The `PolicyRecordDebugLogs` exceeded this limit, leading to errors that blocked new case records from completing. Despite thorough testing in multiple test environments prior to deployment, a misconfiguration in the live environment led to the issue. Unfortunately, this was due to human error in applying the necessary settings correctly during implementation. #### **Resolution** Once the issue was identified, corrective action was taken, and normal service was restored by 13:48 on Sunday. #### **Preventative Measures** To prevent similar issues in the future, we have implemented additional safeguards, including: * Enhanced deployment checks to ensure live configurations match tested environments. * Additional validation steps in our change management process. * Improved monitoring and alerting to detect similar issues earlier and prevent service disruptions. We sincerely regret the disruption caused and take full responsibility for the error. We are committed to improving our processes to ensure that such incidents do not occur again.

Read the full incident report →

Started Jan 06, 2025, 11:57 AM UTC· resolved Jan 06, 2025, 12:54 PM UTC

1. investigating Jan 06, 2025, 11:57 AM UTC

   We are currently investigating an issue when multifactor authentication is triggering for each login, even when then "remember this device" option had been selected. For further updates please see https://status.schemeserve.com/

2. resolved Jan 06, 2025, 12:54 PM UTC

   This incident has been resolved.

Read the full incident report →