Onfido Outage History
Onfido is up right nowPingoru has recorded 52 incidents for Onfido: 36 in the last 12 months and 8 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 3 of 720 (0%) |
| May 2026 | 3 | 16 of 744 (2%) |
| June 2026 | 6 | 53 of 720 (7%) |
| July 2026 | 2 | 4 of 744 (1%) |
| August 2026 | 2 | 19 of 744 (3%) |
| September 2026 | 4 | 16 of 720 (2%) |
| October 2026 | 1 | 4 of 514 (1%) |
Incidents, newest first
Minor October 20, 2026 IDR: provider issue causing delays on reports 3h 35m Identity Enhanced, Identity Enhanced
Started Oct 20, 2026, 06:23 AM UTC· resolved Oct 20, 2026, 09:59 AM UTC
- investigating Oct 20, 2026, 06:23 AM UTC
One of our provider is having issues, which will cause delays on some reports. We're investigating the issue.
- identified Oct 20, 2026, 07:05 AM UTC
Our provider is still experiencing issue. We're closely monitoring the situation. ID-UK is our most impacted service.
- identified Oct 20, 2026, 07:52 AM UTC
The provider is aware of the issue. We're still ready to resume our services as soon as the issue is solved.
- identified Oct 20, 2026, 08:55 AM UTC
Our provider is affected by a major global incident at one of its vendors, which is actively being worked on.
- monitoring Oct 20, 2026, 09:25 AM UTC
Errors stopped. We're monitoring our services to make sure everything is back to normal.
- resolved Oct 20, 2026, 09:59 AM UTC
Our services are operational again.
Minor September 30, 2026 Degraded performance for Identity Reports in UK jurisdiction 3h 6m Identity Enhanced
Started Sep 30, 2026, 10:55 AM UTC· resolved Sep 30, 2026, 02:01 PM UTC
- investigating Sep 30, 2026, 10:55 AM UTC
We are facing issues with our providers and we see Identity Reports not being processed in UK jurisdiction
- identified Sep 30, 2026, 11:20 AM UTC
The issue has been identified and we applied a workaround
- monitoring Sep 30, 2026, 12:35 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Sep 30, 2026, 12:37 PM UTC
We are continuing to monitor for any further issues.
- resolved Sep 30, 2026, 02:01 PM UTC
This incident has been resolved
Major September 22, 2026 UK EVisa Government Service is down 25m UK EVisa
Started Sep 22, 2026, 11:23 AM UTC· resolved Sep 22, 2026, 11:49 AM UTC
- identified Sep 22, 2026, 11:23 AM UTC
This UK Government service is currently down and not available. We are monitoring the situation until the service comes back online.
- identified Sep 22, 2026, 11:24 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Sep 22, 2026, 11:40 AM UTC
3rd party provider appears to be back online now. Checks are being processed. Continuing to monitor the situation.
- resolved Sep 22, 2026, 11:49 AM UTC
3rd party provider is back online. Checks previously stuck have been processed now.
Minor September 7, 2026 Authorization errors in Public API 12h
Started Sep 07, 2026, 02:30 AM UTC· resolved Sep 07, 2026, 02:30 PM UTC
- resolved Sep 07, 2026, 03:53 PM UTC
Between 14:30 and 15:30 UTC, customers in the CA region experienced degraded access to the Public API. A faulty deployment in the authentication layer caused valid tokens to be incorrectly rejected, resulting in failed API requests. The issue has been fully resolved. No action is required on your end. We apologise for the disruption.
Major September 1, 2026 Web SDK Outage (all regions) 24m
Started Sep 01, 2026, 01:21 PM UTC· resolved Sep 01, 2026, 01:45 PM UTC
- resolved Sep 01, 2026, 02:37 PM UTC
From approximately 1:21 PM UTC to 1:46 PM UTC, there was a disruption to Document requests from the Web SDK. Android & iOS SDKs were not impacted. We apologize for this disruption. A detailed postmortem will follow once we've concluded our investigation.
- postmortem Sep 14, 2026, 09:18 AM UTC
### Summary On 1 September 2026, from 13:21 UTC to 13:45 UTC, users attempting document capture in the Web SDK were unable to finish their session. This also extended to native Entrust IDV SDK’s for configurations that contain web modules. Native Onfido Android and iOS SDKs were unaffected. ### Root Causes A routine update to a container image for our document configuration service introduced a regression to the synchronization job that updates the documents list. This resulted in the endpoint that our SDKs use to fetch documents returning `500` errors. This prevented web SDK flows from proceeding. Our native SDKs use a backup document list which allowed them to continue as normal. Our progressive roll-outs are designed to catch service faults before they become significant. However, the synchronization job failure did not mark the readiness probe as unhealthy, so the rollout continued. ### Timeline _13:10 UTC:_ We started a progressive rollout of the change. _13:21 UTC_: The service was fully rolled out leading to errors on impacted SDK functionality. _13:26 UTC:_ Monitoring alerted the team. _13:44 UTC:_ Cause identified and change rolled back. _13:45 UTC:_ Service restored. ### Remedies * We will add missing automated testing coverage to ensure that we catch these cases in the future. * We will add a fallback document list to the Web SDK, similar to the mechanism present in the native SDKs, to ensure it can still function in the case of any similar future failures. * We will fix the progressive rollout to ensure that the rollout will not continue in the case that something has gone wrong with synchronizing documents.
Minor August 26, 2026 Increase in Known Faces consider rate 2h 50m
Started Aug 26, 2026, 03:00 PM UTC· resolved Aug 26, 2026, 05:50 PM UTC
- resolved Sep 08, 2026, 01:24 PM UTC
Known Faces experiencing higher consider rates across all regions.
- postmortem Sep 08, 2026, 01:24 PM UTC
## Summary On August 26, 2026, an increase in the consider rate of Known Faces was observed during the period between 15:00 and 18:00 UTC. The overall observed consider rate of Known Faces increased from ~6% to ~23% across all regions. Perceived increase may have been higher or lower, depending on the client’s specific geography or vertical. The was driven by a change to how matching scores are calculated for Known Faces matches, which inflated scores for non-matching faces and thus causing many reports to return false matches and a **consider** result. ## Root Causes The change introduced a new algorithm for scoring matching faces, which is intended to improve Known Faces recall performance. This was extensively evaluated and validated by internal benchmarks before being released. Inadvertently, the transition from benchmarks to production partially failed to account for particular production environment characteristics. At the root, a single constant wasn’t properly converted in the transition, resulting in wrong behavior of the new scoring algorithm. ## Timeline 15:20 UTC: Change to Known Faces is released 17:35 UTC: Increased consider rate on Known Faces is noticed. 17:50 UTC: Change is rolled back. Consider rate drops soon after. ## Remedies We are reviewing internal benchmarks and testing procedures which can increase the safety of this kind of change. Specifically, ensuring internal testing reflects the real production landscape when it comes to datasets and that transitions from internal benchmarks and experiments to production are thoroughly tested. In addition, change procedures will be carried out with tighter monitoring so it allows us to notice and act faster.
Minor August 24, 2026 Spike in Visual Authenticity failures on Facial Similarity Motion randomness checks 15h 53m
Started Aug 24, 2026, 03:00 PM UTC· resolved Aug 25, 2026, 06:53 AM UTC
- resolved Sep 08, 2026, 01:17 PM UTC
Facial Similarity Motion checks using randomness challenges are failing at a much higher rate than normal in the EU region.
- postmortem Sep 08, 2026, 01:17 PM UTC
## Summary On 24 August 2026, between 15:00 UTC and 06:53 UTC the following morning, Facial Similarity Motion checks using randomness challenges failed at a much higher rate than normal in the EU region. A stricter validation was applied to all customers in the region when it should have been limited to a single customer, causing verifications that would otherwise have passed to be rejected. Affected reports were returned with a "Visual Authenticity" failure. Because Motion checks are fully automated, the end users behind these reports were rejected with no manual review, and some attempted the check again and were rejected a second time. The clear rate for these checks fell from around 97% to around 54% for approximately 1.67% of Facial Similarity Motion reports were affected. Disabling the change restored normal clear rates immediately. Other Motion checks and other regions were unaffected. ## Root Causes A stricter validation of the motion randomness challenge was being trialled with a single customer, controlled by a per-customer configuration setting. A bug in our code meant the setting was evaluated without the customer identifier, so the stricter validation was applied to all customers in the EU region running Motion with randomness checks rather than to the intended one. Verifications that would have passed under the standard validation were rejected instead, and reported under the "Visual Authenticity" breakdown. No customer configuration or submitted data was at fault. ## Timeline All times UTC. * 24 August, 15:00: we enabled a stricter validation for motion randomness challenges in the EU region. Motion checks using randomness began failing at a much higher rate. * 24 August, 16:36: a customer reported a drop in workflow success rates and we began investigating. * 25 August, 04:18: we confirmed the elevated failure rate and escalated it. * 25 August, 06:53: we reverted the change and clear rates returned to normal within minutes. * 26 August: we deployed a permanent fix. ## Remedies * Strengthen the controls around customer-specific configuration changes to ensure they cannot be applied more broadly than intended. * Improve monitoring and alerting for Motion randomness clear rate checks so that unexpected changes in rejection rates are detected automatically and escalated more quickly.
Minor July 27, 2026 Delays in processing reports in the US cluster 1h 9m API, Document Verification, Facial Similarity, Known faces
Started Jul 27, 2026, 06:12 PM UTC· resolved Jul 27, 2026, 07:21 PM UTC
- investigating Jul 27, 2026, 06:12 PM UTC
We are currently investigating delays in our US cluster for the processing of Facial Similarity, Document and Known Faces reports.
- investigating Jul 27, 2026, 06:23 PM UTC
We are seeing degraded performance of some nodes in our search cluster for indexed faces. We continue to investigate the root cause and possible remediations.
- identified Jul 27, 2026, 06:51 PM UTC
We've identified the root cause of degraded performance in our US search cluster and are actively working to resolve it. We'll follow up once fully restored.
- monitoring Jul 27, 2026, 07:18 PM UTC
We implemented a fix and continue to monitor the quality of processing time, which is now improved.
- monitoring Jul 27, 2026, 07:20 PM UTC
Processing time keeps improving, all systems are now operational.
- resolved Jul 27, 2026, 07:21 PM UTC
Processing times are back to normal. This issue is now resolved. Post-mortem to follow soon.
Minor July 24, 2026 Identity Enhanced issues 2h 49m Identity Enhanced, Identity Enhanced
Started Jul 24, 2026, 04:18 AM UTC· resolved Jul 24, 2026, 07:08 AM UTC
- investigating Jul 24, 2026, 04:18 AM UTC
Identity Enhanced is experiencing a major decrease in clear rates for applicants all other the world except UK
- identified Jul 24, 2026, 04:36 AM UTC
We are working with our third-party provider to resolve the issue and will share updates as they become available.
- identified Jul 24, 2026, 04:55 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Jul 24, 2026, 05:09 AM UTC
We are seeing our provider's service recovery and request success rates improving. We continue to monitor the situation closely.
- resolved Jul 24, 2026, 07:08 AM UTC
The provider has resolved the issue, and the service is now operating normally
Critical June 23, 2026 QES tasks cannot complete because our provider is encountering issues. 1h 11m QES
Started Jun 23, 2026, 02:38 AM UTC· resolved Jun 23, 2026, 03:49 AM UTC
- investigating Jun 23, 2026, 02:38 AM UTC
We cannot complete QES tasks because our electronic signature provider is encountering issues.
- identified Jun 23, 2026, 02:57 AM UTC
Our provider is experiencing issues. All QES tasks cannot complete for now.
- identified Jun 23, 2026, 03:20 AM UTC
Our provider is still experiencing issues. All QES tasks cannot complete for now.
- monitoring Jun 23, 2026, 03:45 AM UTC
The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be.
- resolved Jun 23, 2026, 03:49 AM UTC
This incident has been resolved.
Critical June 22, 2026 Severe service disruption across the platform in EU 12m API, Dashboard, Applicant Form, Document Verification and 8 more
Started Jun 22, 2026, 05:44 PM UTC· resolved Jun 22, 2026, 05:56 PM UTC
- monitoring Jun 22, 2026, 05:44 PM UTC
We're currently monitoring the EU cluster after an Amazon RDS issue.
- resolved Jun 22, 2026, 05:56 PM UTC
We’re seeing recovery across our internal metrics, and processing has now returned to full capacity. At this time, the issue appears to be resolved. Our current leading hypothesis is resource contention on a shared Amazon RDS instance that several services depend on, potentially related to a VACUUM operation running alongside a long-running job deleting a large volume of accumulated historical data. We have not yet confirmed the root cause and will continue investigating as follow-up, but service has been restored for now. We'll be following up with a public post-mortem.
- postmortem Jul 02, 2026, 02:27 PM UTC
**Incident date:** 22 June 2026 **Region:** EU \(eu-west-1\) **Affected EU services**: API, Dashboard, Applicant Form, Document Verification, Facial Similarity, Watchlist, Identity Enhanced, Webhooks, Known Faces, Autofill, QES and Device Intelligence. **Customer impact:** ~16:50–17:00 UTC \(acute degradation\); ~17:00–17:20 UTC \(backlog recovery\) ## Summary On 22 June 2026, from approximately 16:50 UTC, a shared database cluster serving our EU region came under severe load and could not reliably serve queries for about 10 minutes. EU services returned elevated errors, and processing throughput briefly fell to ~20–35% of normal levels, with many subcomponents of our system \(e.g., Facial Similarity report processing\) being entirely disrupted, some others less heavily impacted \(e.g., Document report processing\). Service recovered by 17:01 UTC, the database fully stabilizing after an automatic failover \(~17:05–17:07 UTC\). A resultant report backlog was cleared by ~17:20 UTC. Requests in flight during the acute degradation window may have failed unless retried; queued background work was processed automatically once the database recovered. ## Root cause The incident was triggered by a routine database storage-reclamation task following standard scheduled data-deletion processing. This task normally completes without issue; why it failed on this occasion remains under investigation, although we observed that it was processing a larger-than-usual backlog. We have a support case open with our cloud provider to confirm a definitive root cause. The reclamation task began to compete with normal application queries, which slowed as the database struggled to keep up. Applications opened more and more connections, leading to connection saturation and causing queries across the affected services to fail. The database stabilized when an automatic failover to a healthy standby was triggered; the contention fully resolving with the failover to a new instance. ## Timeline \(UTC\) * **16:50** — Our monitoring detected errors and elevated latency across EU services caused by resource contention on a shared database cluster. * **16:53** — We start to see improvements, but system still not acting at normal levels of performance. * **~17:00** — Customer-facing errors subsided and processing resumed as the contention eased. * **17:01** — On-call engineers opened an incident and continued investigations. * **~17:05–17:07** — The database performed an automatic failover to a healthy instance, which reset the overloaded writer and fully stabilised the cluster. The failover was triggered because of resource contention \(out of memory\) caused by the heavy vacuuming in the preceding minutes of the incident. Once the impacting vacuum operations had finished freeing up resources, we had started to see signs of improvement \(16:53—17:01\), but added latency in the feedback loop and aggregation window at AWS still decided to trigger the failover, even though we were already in a recovering state. * **17:15–17:20** — Requests that had queued during the incident were worked through and the backlog returned to normal. * **17:21–17:56** — We monitored the recovery and confirmed processing remained at full capacity. ## Remedies * Reviewing connection limits and pooling so a single service cannot saturate a shared database, and evaluating dedicated database clusters per product to remove cross-service impact. * Changing large historical-data deletions to run in smaller, throttled batches, and tuning database maintenance to avoid large catch-up operations. * Adding earlier, proactive alerting on database memory, connections and load so we can intervene before customer impact. * Continue working with our cloud provider on a definitive root cause.
Minor June 17, 2026 Electronic signature increased TaT 4h 23m QES
Started Jun 17, 2026, 04:59 AM UTC· resolved Jun 17, 2026, 09:23 AM UTC
- identified Jun 17, 2026, 04:59 AM UTC
One of our providers is still experiencing issues, resulting in higher TaT for electronic signature tasks.
- identified Jun 17, 2026, 05:30 AM UTC
Our provider is actively working on fixing the issue.
- identified Jun 17, 2026, 06:19 AM UTC
Our provider is still working on fixing the issue. We're monitoring the impact and we're making sure to keep the TaT as low as possible, considering the situation.
- identified Jun 17, 2026, 06:50 AM UTC
Our provider's error rate dropped significantly. The TaT of QES tasks is now close to the usual value.
- identified Jun 17, 2026, 08:17 AM UTC
We're working closely with our provider to find the reason of the remaining errors. The error rate contacting our provider is stable and the impact on the QES task TaT is under control.
- monitoring Jun 17, 2026, 08:48 AM UTC
Our provider was able to fix the issue and we don't get any error when contacting them. The TaT of QES tasks is back to normal. Monitoring the situation to make sure the errors don't come back.
- resolved Jun 17, 2026, 09:23 AM UTC
The QES tasks TaT is back to normal.
Minor June 16, 2026 Electronic signature high TaT 58m QES
Started Jun 16, 2026, 10:19 PM UTC· resolved Jun 16, 2026, 11:17 PM UTC
- identified Jun 16, 2026, 10:19 PM UTC
Our electronic signature tasks are taking more time to complete because one of our providers is doing some maintenance.
- identified Jun 16, 2026, 10:52 PM UTC
Our provider is still not able to answer to all requests, resulting in longer-than-usual electronic signature task processing time.
- resolved Jun 16, 2026, 11:17 PM UTC
The service is operational again.
Notice June 15, 2026 Sporadic PDF generation issue 2d 1h
Started Jun 15, 2026, 09:30 AM UTC· resolved Jun 17, 2026, 11:00 AM UTC
- resolved Jul 13, 2026, 08:39 AM UTC
We are currently investigating this issue.
- postmortem Jul 13, 2026, 08:40 AM UTC
## Summary Between May and June 2026, around 0.037% of the evidence files generated on the platform were incomplete and included in the corresponding evidence folders. There was no impact on workflow results, and no incorrect information was ever displayed — the affected files were simply empty rather than wrong. A small number of requests to generate timeline files were also affected \(around 0.073%\). ## Root Causes The issue was introduced during an initiative to improve how timeline and evidence PDFs are generated — making them significantly smaller, more efficient, and more resilient to produce. As part of these changes, in rare cases the new flow attempted to generate the PDF before its content had fully loaded, resulting in a blank or nearly empty file. Because the problem was intermittent and occurred only in rare timing conditions, it was not consistently reproduced during rollout testing. ## Timeline * 15 June 2026: Engineering investigation found an incomplete PDF and started the root cause investigation * 17 June 2026: Root cause investigation completed and a safeguard to prevent incomplete PDFs was deployed ## Remedies * We corrected the PDF generation flow so PDFs are only printed after content has fully loaded, and added a safeguard that stops generation when an incomplete PDF is detected. * We also improved our monitoring and test coverage for PDF generation, so similar issues are caught earlier in the future.
Major June 6, 2026 Partial outage for Watchlist reports 1h 41m Watchlist, Watchlist
Started Jun 06, 2026, 02:08 PM UTC· resolved Jun 06, 2026, 03:49 PM UTC
- investigating Jun 06, 2026, 02:08 PM UTC
Some Watchlist search search profiles are not working correctly the impacted reports are not being processed. We are investigation the root cause.
- identified Jun 06, 2026, 02:36 PM UTC
We identified the issue and are implementing a fix.
- monitoring Jun 06, 2026, 03:05 PM UTC
The issue has been fixed, and the reports are being processed correctly. We are still monitoring the service.
- resolved Jun 06, 2026, 03:49 PM UTC
This incident has been resolved.
Critical May 5, 2026 QES tasks outage 1h 46m QES
Started May 05, 2026, 06:52 AM UTC· resolved May 05, 2026, 08:39 AM UTC
- investigating May 05, 2026, 06:52 AM UTC
We noticed an issue with our QES tasks, we're investigating.
- identified May 05, 2026, 06:56 AM UTC
Our provider is experiencing issues. All QES tasks cannot complete for now.
- identified May 05, 2026, 07:35 AM UTC
Our provider is fixing the issue. QES tasks still cannot complete.
- monitoring May 05, 2026, 08:04 AM UTC
The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be.
- resolved May 05, 2026, 08:39 AM UTC
QES is operational again.
Critical April 21, 2026 QES tasks degraded performance 2h 44m QES
Started Apr 21, 2026, 02:28 PM UTC· resolved Apr 21, 2026, 05:12 PM UTC
- identified Apr 21, 2026, 02:56 PM UTC
We identified an issue with one of our provider and some QES tasks may encounter some problems.
- identified Apr 21, 2026, 03:14 PM UTC
Our provider is still experiencing issues. All QES tasks cannot complete for now.
- identified Apr 21, 2026, 04:12 PM UTC
Our provider continues experiencing issues and is working on a fix.
- identified Apr 21, 2026, 05:02 PM UTC
Our provider is slowly recovering, we expect some QES tasks to complete.
- resolved Apr 21, 2026, 05:12 PM UTC
QES is operational again.
Major February 3, 2026 QES tasks partial outage 2h 33m QES
Started Feb 03, 2026, 10:42 AM UTC· resolved Feb 03, 2026, 01:16 PM UTC
- identified Feb 03, 2026, 10:42 AM UTC
One of our provider is experiencing issues. QES capture tasks will fail and QES verification tasks will have an increase in TaT.
- identified Feb 03, 2026, 11:13 AM UTC
The provider is working on a fix, QES is still suffering a partial outage.
- identified Feb 03, 2026, 11:44 AM UTC
The provider is still working on a fix.
- identified Feb 03, 2026, 12:16 PM UTC
The provider is still working on a fix. QES verification tasks may start to time out, depending on their configuration, as we passed 90 minutes of downtime.
- monitoring Feb 03, 2026, 12:46 PM UTC
The provider fixed the issue. We can confirm our QES tasks are now processed correctly. We'll monitor the situation to make sure the error is indeed fixed.
- resolved Feb 03, 2026, 01:16 PM UTC
We can confirm the error stopped and services are stable and operational.
Minor January 28, 2026 Service Degradation - Manual Tasks 4h 6m Document Verification
Started Jan 28, 2026, 11:16 AM UTC· resolved Jan 28, 2026, 03:22 PM UTC
- investigating Jan 28, 2026, 11:16 AM UTC
We are currently investigating this issue.
- monitoring Jan 28, 2026, 11:58 AM UTC
Issue found and fixed. Increased turn around time for manual reports. Estimated time to live manual processing is 4h.
- resolved Jan 28, 2026, 03:22 PM UTC
Incident is fully resolved, manual processing is now working normally. Manual reports will keep having an increased turn around time for a few more hours while it works through the task backlog.
- postmortem Feb 02, 2026, 04:14 PM UTC
### Summary Manual task assignment for all EU customers stopped working between 10h50 UTC and 11:50 UTC. This led to an increase in manual processing Turnaround Time \(TaT\) affecting approximately 20% of our document verification volumes with all customers recovering to TaT SLA by 18h00 UTC. During this period: * All checks that required **manual review** showed an increase in TaT. * **Fully automated reports were not affected** and continued to run as normal. The issue was caused by a **configuration error in our internal task management system**, which prevented it from correctly assigning tasks to our analysts. We fixed the configuration and **restored normal processing** by 28 Jan 2026 11h50 AM UTC, and cleared all manual task backlogs by 18h00 UTC. We have updated our validation and deployment checks to prevent similar issues in the future. ### Root Causes _Manual processing queue assignment was affected by an invalid manual configuration input. This single queue configuration parameter resulted in an error that affected assignments in all queues._ ### Timeline * all times in UTC: _10:50: Configuration manually updated and errors started, no more tasks assigned._ _11:02: On-call is notified of a spike in manual system assignment errors through our monitoring_ _11:07: The error responsible for the spike is identified \(invalid UUID\)_ _11:10: Incident declared_ _11:45: Origin of invalid UUID is found_ _11:50: Bad configuration parameter is deleted_ _11:56: Configuration reintroduced correctly_ _18:00: Recovered from manual task backlog_ ### Remedies * Adding appropriate input configuration value validation * Improve task assignment resilience to these types of errors * Review configuration guidance and post-release monitoring
Major January 26, 2026 Document report processing disrupted in EU 1h 15m
Started Jan 26, 2026, 12:00 PM UTC· resolved Jan 26, 2026, 01:15 PM UTC
- resolved Jan 26, 2026, 12:46 PM UTC
Between 12:15 UTC and 12:40 UTC there was disruption to document report processing in EU. Autofill requests also saw a disruption between 12:16 UTC and 12:27 UTC. A postmortem will follow.
- postmortem Jan 30, 2026, 11:40 AM UTC
## Summary On 26 January 2026 between 12.16 UTC and 12.35 UTC, our document reports processing was severely degraded with customers experiencing extended processing time to about 85% of their traffic. The incident also caused extended processing time on Biometric Authentications and Biometric Verifications for 20% of the traffic between 12.25 UTC and 12.35 UTC. ## Root Causes A core service for fraud prevention on documents came under elevated load, causing kubernetes pods to go down in quick sequence. Our upstream service retry policy proved too aggressive to let the service recover and required manual scaling-up. The retry policy also caused elevated load on a shared database which in turn also affected the biometrics service. ## Timeline * **12:16 UTC** – Our monitoring detected a sharp increase in errors when processing document reports. * **12:19** **UTC** – The on‑call team was alerted and began investigating the affected document‑processing service. * **12:25 UTC** – We identified that the incident was also affecting a small portion of biometric checks, leading to some failures and short delays. * **12:29** **UTC** – On-call engineers manually increased capacity for the impacted document fraud‑prevention service. * **12:35** **UTC** – Error rates for both document reports and biometric checks returned to normal and new requests were being processed successfully. * **12:45–13:06** **UTC** – We processed document reports that were impacted during the incident window. * **13:15** **UTC** – We confirmed that all affected document and biometric requests had completed successfully and marked the incident as resolved. ## Remedies * We have continued to fine-tune our auto-scaling parameters of the affected service to scale up with lower CPU targets. * We modified the retry policy of fraud services to avoid overloading an already struggling service so that it can auto-recover. * We made our load-tests on this service more representative of production traffic \(similar image sizes/document type distribution…\) and test for accelerated traffic spikes. * We lowered the total amount of shared database connections the document processing can take to avoid noisy neighbour impact on biometrics processing.
Critical January 21, 2026 Document Reports not being processed in US 1h 29m Document Verification, Autofill
Started Jan 21, 2026, 11:05 AM UTC· resolved Jan 21, 2026, 12:34 PM UTC
- investigating Jan 21, 2026, 11:05 AM UTC
We have a high error rate in document report processing and autofill. Our team is investigating
- identified Jan 21, 2026, 11:49 AM UTC
We have identified the issue as a configuration issue between services, and are working on a fix.
- monitoring Jan 21, 2026, 12:07 PM UTC
We have deployed a fix. New document reports and autofill requests are being processed normally. We will rerun the earlier impacted reports.
- resolved Jan 21, 2026, 12:34 PM UTC
This issue is now resolved: Between 10:30 and 12:00 UTC, document reports could not be processed and autofill requests were failing in the US region. Starting from 12:00, live traffic was unaffected. From 12:00 to 12:30 UTC, pending reports were processed. We take a lot of pride in running a robust, reliable service and we're working hard to make sure this does not happen again. A detailed postmortem will follow once we've concluded our investigation.
- postmortem Jan 30, 2026, 11:39 AM UTC
### Summary Around 10:29 UTC, around 10 minutes after restoring service in the EU and CA from the 10:05-10:17 UTC outage, we go alerted for high error rate in the US region and we noticed the new ML model version was back online in that region. This impacted Turnaround Time \(TaT\) for all Document Report and Studio Autofill until 12:11 UTC. Autofill Classic had an average 100% error rate during the same time frame. All impacted Document Reports and Studio Autofill tasks were completed successfully with an average TaT of ~40 minutes. ### Root Causes A previously rolled back version of an ML model went back live automatically in the US region. The automated canary reversal did not succeed in that region, leaving the deployment in an inconsistent state. As a result, the model routing labels were not updated as expected during the next forced deployment, and traffic continued to be served by pods running the incorrect model version until we manually rolled back the model in the cluster. This was the result of a bug in our version of Helm. ### Timeline _10:29 UTC: The new model release went back live in the US region only, after the previous automated canary rollback_ _11:44 UTC: We manually re-deployed a previous version via CI/CD pipeline. This was not successful and errors continued._ 1_2:00 UTC: We manually rolled back the version of the model directly in the cluster._ 1_2:11 UTC: All services went back to normal_ ### Remedies * We will fix our deployment tooling to reliably apply model-routing label changes to these resources by upgrading Helm.
Notice January 21, 2026 Delays in Document report processing (all Regions) 1h
Started Jan 21, 2026, 09:00 AM UTC· resolved Jan 21, 2026, 10:00 AM UTC
- resolved Jan 21, 2026, 10:25 AM UTC
Between 10:05 UTC and 10:18 UTC we experienced delays for a significant part of document reports, as well as errors on our Autofill product. This was due to the release of a faulty component. The release was rolled back to resolve the situation. The delayed reports backlog is being processed.
- postmortem Feb 03, 2026, 07:14 PM UTC
### Summary The combined deployment of a new Document Extraction ML model and respective service that support Document Verification and Autofill, replaced the previous model version instantly, while rolling out the application layer via canary. When we got alerted for errors during the canary release, we triggered a manual rollback \(both application layer and ML model\), which failed due to a bug in the model deployment framework. Subsequently, the canary release for the application layer failed to progress due to high error rate and automatically reverted both the application and model layer reverting the service to a healthy state. The service unavailability resulted in increased Turnaround Time \(TaT\) for all Document reports and Studio Autofill created between 10:05 and 10:17 UTC. Autofill Classic had an average 70% error rate during that time frame. All impacted Document Reports and Studio Autofill tasks were completed successfully with an average TaT of ~6 minutes. ### Root Causes Our model deployment system doesn’t support model version replacement without downtime. Application and model pods are deployed without synchronization or any specific order requiring a 3-step release \(add new version, update application layer, remove old version\). The lack of automated guardrails preventing releases to replace a model version in a single step has led to the service unavailability due to human error. Our rollback mechanism for models didn’t restore correctly the model config map in the Kubernetes cluster, which led the rollback procedure to fail for both application and models when triggered manually. It succeeded during canary automated reversal. ### Timeline _10:05 UTC: Production release of extraction service updating an ML model_ _10:08 UTC: We unsuccessfully_ _tried to manually rollback after noticing errors on a metrics dashboard_ _10:10 UTC: Automatic monitors alerted the On-call team due to high error rate_ _10:16 UTC: The canary release was aborted automatically due to high error rate_ _10:18 UTC: All services went back to normal_ ### Remedies 1. Add guardrails to the release pipeline in order enforce a 3-step safe release process \(new ML model deployment, application layer update, ML model removal\) 1. We will separate CI/CD pipelines for ML model deployment and application layer 2. Model switching will be exclusively done on the application layer 2. Fix the manual rollback mechanism for models, in particular for releases where we remove a model version
Notice January 14, 2026 High Processing times for Video Facial Similarity Checks –
Started Jan 14, 2026, 01:30 AM UTC· resolved Jan 14, 2026, 01:30 AM UTC
- resolved Jan 20, 2026, 02:20 PM UTC
Video facial similarity tasks are taking longer than expected to complete
- postmortem Jan 20, 2026, 02:21 PM UTC
### Summary _An increase of Facial Similarity Video Manual tasks backlog resulted in higher processing times for about 21% of that variant’s traffic._ ### Root Causes _A spike in Biometric Video volumes overwhelmed our BPO resources, lengthening Facial Similarity Video check turnaround times impacting around 20% of variant traffic._ ### Timeline _01:16 GMT: A significant backlog of tasks was reported._ _03:27 GMT: BPO team allocated more analysts to process pending video tasks._ _06:33 GMT: Backlog was cleared._ ### Remedies Not applicable: the increase in TaT resulted exclusively from an unexpected spike in volume that created a backlog for manual analysts. There were no system issues.
Major December 19, 2025 Errors when processing checks in Canada region 25m API, Document Verification
Started Dec 19, 2025, 10:35 AM UTC· resolved Dec 19, 2025, 11:00 AM UTC
- identified Dec 19, 2025, 10:35 AM UTC
The issue has been identified and we are proceeding to the fix. We will provide an update in 15 minutes.
- identified Dec 19, 2025, 10:39 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Dec 19, 2025, 10:52 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Dec 19, 2025, 11:00 AM UTC
This incident has been resolved.
Major December 18, 2025 Degraded performance of QES and Evidence File products 1h 5m QES
Started Dec 18, 2025, 11:26 AM UTC· resolved Dec 18, 2025, 12:31 PM UTC
- monitoring Dec 18, 2025, 11:26 AM UTC
Our QTSP provider is currently experiencing technical issues that are impacting the performance of our Qualified Electronic Signature (QES) and Evidence File services. We are actively monitoring the situation and will provide updates as soon as more information becomes available.
- monitoring Dec 18, 2025, 11:49 AM UTC
Performance of the Evidence File service has been fully restored. However, the Qualified Electronic Signature (QES) service continues to experience significant degradation due to ongoing issues on the QTSP provider’s side. We are actively monitoring the situation and will share further updates as soon as possible.
- resolved Dec 18, 2025, 12:31 PM UTC
The incident was resolved