Onfido Outage History

Started Dec 18, 2025, 11:26 AM UTC· resolved Dec 18, 2025, 12:31 PM UTC

1. monitoring Dec 18, 2025, 11:26 AM UTC

   Our QTSP provider is currently experiencing technical issues that are impacting the performance of our Qualified Electronic Signature (QES) and Evidence File services. We are actively monitoring the situation and will provide updates as soon as more information becomes available.

2. monitoring Dec 18, 2025, 11:49 AM UTC

   Performance of the Evidence File service has been fully restored. However, the Qualified Electronic Signature (QES) service continues to experience significant degradation due to ongoing issues on the QTSP provider’s side. We are actively monitoring the situation and will share further updates as soon as possible.

3. resolved Dec 18, 2025, 12:31 PM UTC

   The incident was resolved

Read the full incident report →

Started Dec 11, 2025, 11:12 AM UTC· resolved Dec 11, 2025, 03:42 PM UTC

1. resolved Dec 18, 2025, 09:25 AM UTC

   A recently enabled feature in our services caused an unexpected increase in source integrity breakdown rejections on Facial Similarity reports, due to payload integrity verification (used to flag network injection attacks). This anomaly resulted in 5K reports being incorrectly flagged as “consider” for customers using the Standard and Photo Fully Auto variants of Facial Similarity reports, and only for clients that leverage the Device Intelligence report.

2. postmortem Dec 18, 2025, 09:26 AM UTC

   ### Summary A recently enabled feature in our services caused an unexpected increase in source integrity breakdown rejections on Facial Similarity reports, due to payload integrity verification \(used to flag network injection attacks\). This anomaly resulted in 5K reports being incorrectly flagged as “consider” for customers using the Standard and Photo Fully Auto variants of Facial Similarity reports, and only for clients that leverage the Device Intelligence report. ### Root Causes We’ve determined that the root cause of the issue was a race condition bug on the live photo upload module of the Smart Capture Web SDK which, for clients having the Device Intelligence product enabled, could inadvertently cause incorrect signatures to be computed. ### Timeline **2025-12-10 11:12 GMT**_:_ A code change was rolled out to start collecting new device intelligence data for added fraud prevention. **2025-12-11 10:50 GMT**: High payload integrity flagging was identified, signalling an apparent rise in network injection attacks, which turned out to be a false flag for fraud, but a true signal for something going wrong with our deployment. We took longer than usual to identify the issue due to both a misconfiguration on our alerting stack \(we were wrongfully not alerting on-call personnel for such alerts\), and also the low amount of affected reports given the progressive rollout strategy we took to incrementally deliver this feature. In order to reduce impact, we targeted only a portion of our latest Web SDK version traffic, additionally limiting to the Standard and Photo Fully Auto variants of Facial Similarity reports, and only for clients that leverage the Device Intelligence report. **2025-12-11 12:00 GMT**: The code change and used feature flags for rollout were reverted. **2025-12-11 16:52 GMT**: All the affected reports were rerun. **2025-12-15 15:42 GMT**: After a lengthy investigation that revealed the problem to be a race condition, a fix was implemented \(not yet deployed\). ### Remedies * As a mitigation measure, affected reports were rerun. * We have additionally fixed the misconfiguration of our alerting system to guarantee quick response by on-call personnel next time a similar issue occurs for the payload integrity check. * The implemented fix will soon be released as a way to provide robust payload integrity checking on our Web SDK, resilient to nuanced changes on the device intelligence data collected during a capture session.

Read the full incident report →

Started Dec 09, 2025, 01:33 PM UTC· resolved Dec 09, 2025, 01:46 PM UTC

1. identified Dec 09, 2025, 01:33 PM UTC

   We accumulated a large backlog of messages to update OpenSearch and we are currently processing it. The backlog should be cleared in the next hours.

2. resolved Dec 09, 2025, 01:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 09, 2025, 11:15 AM UTC· resolved Dec 09, 2025, 12:12 PM UTC

1. identified Dec 09, 2025, 11:39 AM UTC

   We have identified an issue in the infrastructure that handles reports creations. We are working on restoring it.

2. monitoring Dec 09, 2025, 11:46 AM UTC

   The API is now fully available. We are monitoring the situation.

3. resolved Dec 09, 2025, 12:12 PM UTC

   The incident has been resolved. We'll share a public post-mortem later.

4. postmortem Dec 19, 2025, 01:19 PM UTC

   ### Summary On **December 9, 2025**, between **09:50 AM UTC and 11:36 AM UTC**, our services in the **EU region** experienced a **degradation** that affected customers’ ability to **list checks in the dashboard**. During this period, the **webhook logs displayed in the dashboard** were also impacted, leading to **incomplete or delayed visibility**, and the **webhook resend feature** was similarly degraded. Furthermore, as a consequence of the ongoing incident, between **11:15 AM UTC and 11:38 AM UTC \(a total of 23 mins\)**, our **Classic** clients were affected by an issue in the check creation service that resulted in **no new checks being created** during that period in the **EU region**. In addition, both **Classic** and **Studio** clients experienced **increased turnaround times \(TaT\)** for ongoing checks and tasks. ### Root Causes The incident occurred because a **shared indexing component** became unavailable after reaching its **capacity limits** due to a workload spike from another internal system. We had prior spikes with more than double the load, lasting longer, that caused no issues. We suspect a timing issue with background maintenance \(e.g., garbage collection\) caused this. However, we lack sufficient historical telemetry to confirm. This overload caused the shared infrastructure to fail, which **cascaded into disruptions across multiple dependent services** and **led to errors in client-facing operations**. ### Timeline 09:50 UTC: Error rate and round trip time increased for an indexing component 09:51 UTC: Our on-call team is notified about an increase in error rate 10:10 UTC: Investigation and actions begin to reduce non-essential traffic and speed recovery 11:08 UTC: Actions to reduce non-essential traffic completed 11:15 UTC: Shared caching infrastructure dependency becomes unavailable 11:15 UTC: Incident impacts check creation; no new checks succeed 11:18 UTC: Upscale indexer as actions to reduce load were insufficient 11:32 UTC: Upscale shared caching infrastructure to resolve checks creation incident 11:36 UTC: Checks and webhook logs show working again on dashboard; resend webhook function still shows outdated information 11:39 UTC: Our on-call team confirms all traffic returned to normal 12:45 UTC: Upscale completed 13:05 UTC: We continued monitoring to ensure backlog items processed correctly 13:20 UTC: Remaining backlog processed ### Remedies * Introduce rate-limiting, review timeouts, and change query access patterns sent to the indexing service to reduce the risk of excessive load events * Extend our monitoring on the shared caching infrastructure to anticipate resource capacity exhaustion; * Review and reduce the checks creation service's dependency on shared caching infrastructure.

Read the full incident report →

Started Dec 09, 2025, 10:00 AM UTC· resolved Dec 09, 2025, 11:30 AM UTC

1. resolved Dec 09, 2025, 12:03 PM UTC

   The incident has been resolved. We'll update the incident message with a post-mortem later.

Read the full incident report →

Started Dec 02, 2025, 12:00 PM UTC· resolved Dec 02, 2025, 01:00 PM UTC

1. resolved Dec 02, 2025, 01:30 PM UTC

   We have experienced delays and errors in check creations in EU Region, as well as inability to log into the dashboard from 13:05 to 13:25 UTC. The incident is now resolved and we are processing traffic. We will write a post-mortem with additional information.

2. postmortem Dec 04, 2025, 08:25 PM UTC

   ### Summary We were in the process of a routine upgrade to our network ingress controller from the “prior” version to a “next” version. This component is packaged and applied to API-related services as and when those services are deployed. A problem occurred when we packaged the “next” version while one of those services was in the middle of a deployment. This problem rendered our external API, as well as our Dashboard, inaccessible in the EU region, for 25 minutes. The issue was as follows: 1. At the start of the service deployment pipeline, the system set the network component dependency to the “prior” version \(current at the time\), and dynamically created configuration settings based on that version. 2. Meanwhile, the “next” version was released while the service deployment pipeline was still in progress. 3. Later in the pipeline, another step overrode the dependency version to "latest” \(now pointing to “next”\). However, the original configuration settings created in the pipeline were incompatible with this new version, which resulted in a corrupted final network configuration. This situation had not previously occurred because the network ingress controller had never changed while an API service was mid deployment. ### Root Causes The root causes were: * The final network configuration state depending on the output of multiple pipeline steps. * The dependency version not being pinned for the entire deployment process, allowing a mid-pipeline version change. ### Timeline 12:39 UTC: An API service deployment pipeline starts 12:56 UTC: The new network ingress controller is released 13:04 UTC: The API service pipeline triggers an EU Production deployment 13:05 UTC: A corrupted ingress configuration is installed, preventing traffic from being routed to the external API 13:10 UTC: We receive monitoring alerts for the loss of API traffic 13:15 UTC: Incident response mobilized and investigating the problem 13:22 UTC: API service rollback is initiated 13:25 UTC: The rollback for the API completed, restoring a valid ingress configuration and allowing traffic again ### Remedies The following improvements will be made to the standard deployment pipeline to prevent similar network configuration inconsistencies from occurring: 1. Use explicit version numbers for dependencies instead of "latest" during deployments. 2. Consolidate the steps required to build the network configuration.

Read the full incident report →

Started Nov 14, 2025, 01:22 PM UTC· resolved Nov 14, 2025, 02:10 PM UTC

1. identified Nov 14, 2025, 01:22 PM UTC

   The issue has been identified and a fix is being rolled out. We will provide a further update in 30 minutes.

2. monitoring Nov 14, 2025, 01:48 PM UTC

   The issue has been fixed for new checks going forward. We are still working on assessing the impact for the completed checks. We will provide a further update in 30 minutes.

3. resolved Nov 14, 2025, 02:10 PM UTC

   This issue is now resolved: We take a lot of pride in running a robust, reliable service and we're working hard to make sure this does not happen again. A detailed postmortem will follow once we've concluded our investigation.

4. postmortem Nov 18, 2025, 04:26 PM UTC

   ### Summary A change was deployed with a bug that resulted in ~30% of all Document Video Reports during a 3 hour period to be falsely rejected for Invalid Media Signature. ### Root Causes This incident was caused by a change deployed to our backend systems, that started incorrectly marking reports with unsigned medias as having incorrect signatures. ### Timeline 09:52 GMT: A change is deployed incorrectly marking unsigned medias as having invalid signatures 12:33 GMT: The team that owns the service is alerted of the issue 12:54 GMT: The change in media signatures handling is identified as the root cause 13:07 GMT: The service is rolled back, and the error stops ### Remedies We are refining our monitoring for the relevant service, so that issues similar to this one will be resolved in a more timely manner. We are reviewing our end to end test suite to ensure consistency in media signature verification.

Read the full incident report →

Started Oct 20, 2025, 02:48 PM UTC· resolved Oct 20, 2025, 08:22 PM UTC

1. identified Oct 20, 2025, 02:48 PM UTC

   We are experiencing degraded performance with QES and OTP services due to an ongoing issue involving AWS and our OTP provider. Our teams are actively working to resolve the situation and restore optimal functionality as soon as possible.

2. resolved Oct 20, 2025, 08:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 01:45 PM UTC· resolved Oct 20, 2025, 08:19 PM UTC

1. investigating Oct 20, 2025, 01:45 PM UTC

   We are currently investigating this issue.

2. investigating Oct 20, 2025, 01:54 PM UTC

   Continued issues with our hosting provider, AWS, have resulted in recurring problems and our US service is experiencing degraded performance. We continue to investigate this issue.

3. identified Oct 20, 2025, 02:29 PM UTC

   The incident has regressed, and we are now facing another outage. Since 14:13 UTC, we have not been able to receive requests or process reports. We continue to monitor the situation with AWS.

4. identified Oct 20, 2025, 03:13 PM UTC

   In the latest update from AWS, multiple services continue to be impacted. In particular, network connectivity issues are still preventing us from servicing requests. We continue to monitor the situation.

5. monitoring Oct 20, 2025, 05:22 PM UTC

   Some AWS services are showing signs of intermittent recovery, but disruption remains widespread and our services remain critically impacted. We continue to monitor the situation.

6. monitoring Oct 20, 2025, 06:48 PM UTC

   AWS are reporting progressive improvements. We are intermittently processing a limited number of transactions, but our services remain unstable.

7. monitoring Oct 20, 2025, 07:32 PM UTC

   Report processing has recovered over the past 40 minutes, and all services are stabilizing. We are continuing to monitor.

8. resolved Oct 20, 2025, 08:19 PM UTC

   This incident is now resolved. We apologize for the disruption caused. A detailed postmortem will follow once we've concluded our investigations.

9. postmortem Oct 24, 2025, 09:31 AM UTC

   # Summary On October 20th, our US regional instance experienced an outage over two distinct periods: from 06:48 UTC to 09:27 UTC, and again from 14:12 UTC to 18:43 UTC \(fully normalizing by 19:06 UTC\). This was caused by a series of major service disruptions with our infrastructure provider, AWS. During these periods, we were unable to accept and process reports, and access to supporting systems, like Dashboard, were mostly unavailable. # Root Causes This incident was caused by a large scale AWS outage in US-EAST-1, the region where we host our US instance. The problem began with a networking failure internal to AWS that led to cascading failures across a wide range of AWS services across the region. Unfortunately, the issue impacted all available zones \(AZ\) in the region, which rendered our multi-AZ redundancy ineffectual. # Timeline ## First outage period 06:48 UTC: API traffic into the US instance stops. No checks or workflows are being created for customers in the affected region. 06:53 UTC: Multiple on-call teams begin receiving alerts of issues in the US region. 07:07 UTC: On-call responders note errors with various internal components, including applications having problems connecting to database services. 07:16 UTC: We confirm an operational issue with our infrastructure provider, as reported on the AWS Service Health Dashboard at 07:11. The incident report refers to _“increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region”_, the region where we host our US instance. API Gateway is one of the many impacted services, which explains why we are not receiving any inbound traffic. 07:17 UTC: Attempts to create a public Incident are unsuccessful due to StatusPage itself being unavailable, being impacted by the same AWS problem. On-call response subsequently notifies internal customer support to make them aware. 07:21 - 09:00 UTC: Attempts to address issues with specific services, such as DynamoDB, RDS and SQS, all fail as we are unable to reliably access and administer the services. We continued to monitor progress with AWS, as well as follow internal monitors for any signs of recovery. 09:01 UTC: AWS report that they have identified a potential root cause for the incident, relating to internal networking services. This initially impacted DynamoDB but this, in turn, led to the cascade of failures across many AWS services because many of these other services themselves depend on DynamoDB. 09:22 UTC: AWS reports that they have applied initial mitigations. Shortly after, we observe signs of recovery, with inbound traffic resuming and backend services stabilizing. 09:27 UTC: Our services are once again operational and incident response transitions into post-resolution monitoring. 09:53 UTC: We publicly report the Incident after StatusPage becomes available. 10:07 UTC: Having closely monitored our systems for another 40 minutes since systems stabilized, our on-call team declares the incident closed. ## Second outage period 13:39 UTC: Our on-call team begins to receive alerts of elevated errors in some internal services in the US region. Upon subsequent investigation we find that our cluster is experiencing failures when attempting to provision new EC2 nodes. This is adding strain to the system, as we are entering morning hours in the US and our internal services are prevented from auto-scaling up due to unavailable capacity. These issues are not yet resulting in significant customer impact, but latency of processing is increasing. 13:45 UTC: We publish a incident on our StatusPage alerting customers to the recurrence of issues in the US instance. 14:12 UTC: The issues escalate significantly, and our system stops processing reports. In the following couple of minutes, we also stop receiving new API requests. 14:14 UTC: AWS confirms significant API errors and connectivity issues across multiple services in the US-EAST-1 region. 15:03 UTC: AWS report they have identified the root cause as an issue originating from within the EC2 internal network, which corresponds with the EC2 provisioning failures that we continue to experience. We attempt to manual provision new nodes, but this also fails. 16:13 UTC: AWS shared that additional mitigation steps were being applied to aid recovery. 16:55 UTC: We begin receiving API traffic again, albeit with intermittent disruptions. However, backend systems remain unstable. 17:03 - 17:38 UTC: AWS continue to apply mitigation steps and begin applying a fix for EC2 launch failures, rolling this out one AZ at a time. 17:22 UTC: Our team continue to see progressive signs of recovery but some instability remains. A small rate of reports are being processed. 18:08 UTC: API traffic normalizes and report processing rates begin to increase. 18:43 - 19:06 UTC: Report processing rates begin to increase rapidly over this period, starting from approximately 50% of normal levels, back up to normal levels. 19:15 UTC: AWS provides an update confirming recovery across all AWS services, and instance launch having normalized across multiple AZs. 19:32 UTC: After seeing sustained stability over the past 30-40 minutes, our on-call team update StatusPage to report that our service has recovered. 20:19 UTC: After a further period of monitoring, we declare the incident closed.

Read the full incident report →

Started Oct 20, 2025, 12:10 PM UTC· resolved Oct 20, 2025, 01:07 PM UTC

1. investigating Oct 20, 2025, 12:10 PM UTC

   We are currently investigating an increase in turnaround time for document check processing in the EU region. We will provide a status update in the next 15 minutes.

2. identified Oct 20, 2025, 12:20 PM UTC

   The issue has been identified and a fix is being implemented. We will provide a further update in 15 minutes.

3. monitoring Oct 20, 2025, 12:38 PM UTC

   We have implemented a fix for this issue. We are monitoring closely to make sure the issue has been resolved and everything is working as expected. Please bear with us while we get back on our feet, and we appreciate your patience during this incident. We will provide an update in the next 30 minutes.

4. resolved Oct 20, 2025, 01:07 PM UTC

   This issue is now resolved: Increased turnaround time for document check processing in the EU region We take a lot of pride in running a robust, reliable service and we're working hard to make sure this does not happen again. A detailed postmortem will follow once we've concluded our investigation.

5. postmortem Oct 22, 2025, 02:16 PM UTC

   ### Summary Between 11:47 and 12:10, 55% of documents reports could not be processed due to a fraud detection service partial failure. Starting 12:10 onwards, traffic was being processed as usual and we started rerunning failed reports. Reports that required manual processing saw additional delays of up to 2 hours in order to clear the backlog. ### Root Causes A sudden increase in CPU usage by the impacted fraud service lasted for a few minutes, leading to retries policies being initiated. The service did not manage to scale properly to handle both the ongoing traffic and the retries, leading to a portion of report processing being halted and stored in dead-letter queues for processing after the systems stabilize. ### Timeline 11:47 UTC: High CPU usage usage on a fraud detection service leads to errors 11:50 UTC: CPU usage is back to normal, reports that errored out are being retried 11:51 UTC: the fraud detection service doesn’t manage to scale to manage both the normal traffic and retries 11:51 UTC: on-call team is alerted of a high error rate on the fraud detection service and starts investigating 12:09 UTC: on-call team identifies the root cause and scales up the service manually 12:10 UTC: Errors have stopped and reports are processed normally 12:12 UTC: We start the process of rerunning reports that failed during the incident 13:20 UTC: All reports that did not require manual review are now completed 13:50 UTC: All reports that required manual review have been processed ### Remedies Review the autoscaling capabilities of the impacted service as well as other services that share a similar architecture.

Read the full incident report →

Started Oct 20, 2025, 09:57 AM UTC· resolved Oct 20, 2025, 12:28 PM UTC

1. identified Oct 20, 2025, 09:57 AM UTC

   The OTP provider is currently impacted by an AWS outage, resulting in errors within QES workflows.

2. monitoring Oct 20, 2025, 09:58 AM UTC

   The OTP provider is currently impacted by an AWS outage, resulting in errors within QES workflows.

3. monitoring Oct 20, 2025, 11:57 AM UTC

   The OTP provider is currently facing ongoing performance issues, which are significantly impacting QES workflows.

4. resolved Oct 20, 2025, 12:28 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 06:45 AM UTC· resolved Oct 20, 2025, 10:07 AM UTC

1. investigating Oct 20, 2025, 09:46 AM UTC

   Due to an outage with our cloud provider we were not able to receive check creation requests or process checks since shortly after 06:48 UTC. The impact was specific to the US region. The same issue impacted StatusPage and delayed publishing this notice. Check processing has since resumed. We are monitoring and assessing the situation.

2. monitoring Oct 20, 2025, 09:46 AM UTC

   Services have been operational since 9:26 UTC. We are monitoring the recovery.

3. monitoring Oct 20, 2025, 09:47 AM UTC

   Systems are stable. We continue to monitor.

4. resolved Oct 20, 2025, 10:07 AM UTC

   The incident is resolved. The root cause was an issue with AWS in the US region that impacted many AWS services. We will prepare and publish a post mortem in the coming days.

Read the full incident report →

Started Oct 09, 2025, 01:17 PM UTC· resolved Oct 09, 2025, 01:55 PM UTC

1. investigating Oct 09, 2025, 01:17 PM UTC

   We've identified an increase in webhooks latency affecting the EU region.

2. monitoring Oct 09, 2025, 01:50 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 09, 2025, 01:55 PM UTC

   This incident has been resolved.

4. postmortem Oct 13, 2025, 02:04 PM UTC

   ### Summary Webhooks latency increased to up to 49 minutes in the EU region between 12:39 and 13:39 UTC on the October 9, 2025. Although no webhooks were lost during the incident, some clients encountered rate limit errors when calling the API while processing a surge of delayed webhooks. These webhooks were subsequently retried according to the logic outlined in our public documentation. ### Root Causes An infrastructure dependency that helps reduce duplicate delivery of webhooks experienced a hardware failure. As a result there was a significant decrease of the service throughput during the incident. ### Timeline 12:39 UTC: Throughput of the service responsible for delivering webhooks decreased 12:45 UTC: Our on-call team gets notified and starts the investigation 13:16 UTC: The on-call team acknowledge the widespread impact of the incident and updates the status page 13:42 UTC: The on-call team identifies the faulty infrastructure dependency 13:49 UTC: The service recovered and all pending webhooks were delivered ### Remedies We’ll improve the resilience of webhook delivery in case of failures to this piece of infrastructure. We will also update our runbooks with specific instructions to help diagnose this type of failure and therefore decrease the recovery time in the case of similar incidents.

Read the full incident report →

Started Oct 09, 2025, 08:35 AM UTC· resolved Oct 09, 2025, 12:54 PM UTC

1. investigating Oct 09, 2025, 09:41 AM UTC

   TaT of Identity Enhanced are affected and could be lower than usual. We are in contact with our providers in order to resolve this issue as soon as possible

2. investigating Oct 09, 2025, 09:42 AM UTC

   We are continuing to investigate this issue.

3. identified Oct 09, 2025, 09:45 AM UTC

   We are in contact with our providers in order to resolve this issue as soon as possible

4. identified Oct 09, 2025, 10:36 AM UTC

   Our provider is working on fixing the issue.

5. identified Oct 09, 2025, 11:29 AM UTC

   The issue is identified, and the provider is working on the fix, but does not provide an ETA yet.

6. identified Oct 09, 2025, 12:28 PM UTC

   Our provider has identified the issue and is actively working on a fix.

7. resolved Oct 09, 2025, 12:54 PM UTC

   The issue on the provider's side was resolved. All the affected reports will be fixed.

Read the full incident report →

Started Sep 27, 2025, 07:45 AM UTC· resolved Sep 27, 2025, 11:17 AM UTC

1. identified Sep 27, 2025, 09:50 AM UTC

   We are facing a partial outage with our provider which leads to increased turnaround time QES tasks

2. identified Sep 27, 2025, 10:44 AM UTC

   We are still facing elevated error rates with our provider but TaT has improved and the issue continues be to monitored

3. monitoring Sep 27, 2025, 11:16 AM UTC

   Our provider recovered and the turnaround times are back to normal values.

4. resolved Sep 27, 2025, 11:17 AM UTC

   The incident has been resolved. All impacted requests during the period of instability were processed through our retry mechanism.

Read the full incident report →

Started Sep 24, 2025, 02:57 PM UTC· resolved Sep 24, 2025, 05:29 PM UTC

1. identified Sep 24, 2025, 02:57 PM UTC

   We have identified an issue in our EU region that is negatively impacting the document report TaT. A fix has been implemented, and we are currently monitoring its effectiveness.

2. monitoring Sep 24, 2025, 03:37 PM UTC

   We have implemented a fix for this issue. We are closely monitoring to ensure the issue has been resolved and everything is working as expected. Please bear with us while we get back on our feet, and we appreciate your patience during this incident.

3. monitoring Sep 24, 2025, 04:36 PM UTC

   The fix has proven to be effective, and we are now processing live document reports within the expected TaT, given that processing is automatic-only. For document reports that require manual processing, we are clearing the backlog as quickly as possible and hope to return to normal TaT within the next 3 hours. Please bear with us while we get back on our feet, and we appreciate your patience during this incident.

4. resolved Sep 24, 2025, 05:29 PM UTC

   This issue is now resolved: Increased document report TaT in the EU cluster. A modest number of the queued manual reports remain and are expected to be cleared soon. We take great pride in running a robust and reliable service, and we're working hard to ensure this does not happen again. A detailed postmortem will follow once we've concluded our investigation.

5. postmortem Sep 27, 2025, 07:31 AM UTC

   ### Summary A change to document classification logic led to 30% of the reports not being automatically classified. Those reports were sent for manual review, leading to an increase in turn around time. The change was reverted shortly after and automated processing returned to normal. Impacted checks were re-run, leaving a residual backlog of reports requiring manual review that was cleared a few hours later. ### Root Causes A change was deployed with a minor error that was not apparent in testing. This led to the model being unable to classify documents reliably, resulting in a significant portion of the reports being sent for review. ### Timeline 13:45 UTC: We deploy a change to our document classification logic. A signifiant portion of traffic starts being sent for manual review. 14:30 UTC: We are notified about a drop in automation rate and increase in manual queues. 14:37 UTC: Having identified the cause and reverted the change, automation rate returns to normal levels. 15:15 UTC: We continue to monitor. Impacted reports are rerun to assist clearing down manual backlog. 17:29 UTC: All reruns completed, positively impacting manual queues. ### Remedies We are adding new monitoring capabilities on automation rates for a faster notification and response time.​ We are reviewing our recovering process so we are able to perform reruns of impacted reports faster. We are adding non regression tests to the classification logic to prevent similar incidents.

Read the full incident report →

Started Sep 23, 2025, 10:08 AM UTC· resolved Sep 23, 2025, 10:40 AM UTC

1. monitoring Sep 23, 2025, 10:08 AM UTC

   We have rolled back a faulty deployment that caused device intelligence reports to be withdrawn. We are currently re-running them. Studio workflow runs that are in progress will be resumed once the device intelligence reports are re-run. This disruption lasted for 1 hour between 07:04 UTC and 08:06 UTC.

2. resolved Sep 23, 2025, 10:40 AM UTC

   All withdrawn reports were re-run. In progress workflow runs should be resumed now.

3. postmortem Oct 06, 2025, 08:44 AM UTC

   ### Summary All Device Intelligence reports created on the 23rd of September 2025 between 06:59 and 08:06 UTC were withdrawn. Moreover, as a consequence the corresponding Studio Tasks did not complete during the incident. Once the root cause was identified and fixed, all withdrawn reports were re-processed and the results were delivered through Webhooks or can be fetched through the Public API. ### Root Causes A faulty deployment in the service that processes the results for this report/task type was released at 06:59 UTC. ### Timeline 06:59 UTC: The service that processes device intelligence reports starts to be deployed into production in all regions. 07:05 UTC: Deployment finishes with success 07:08 UTC: First responders are notified about a slight increase in error rate in Studio tasks 07:21 UTC: First responder starts looking into the issue, without clear indication the deployment was the cause of the error rate increase 07:50 UTC: We understand that 100% of device intelligence reports were withdrawn, showing a larger issue than just an error rate increase in Studio 08:06 UTC: Roll back the service that is responsible for processing this type of report completed and error rate recovers 09:05 UTC: Start preparing to reprocess withdrawn reports 09:44 UTC: Report reprocessing started 10:37 UTC: Report reprocessing finished ### Remedies * Improve the alerting in the service for this report type * Add additional e2e test coverage for the service

Read the full incident report →

Started Sep 17, 2025, 08:34 PM UTC· resolved Sep 17, 2025, 09:16 PM UTC

1. investigating Sep 17, 2025, 08:34 PM UTC

   We are currently investigating this issue.

2. monitoring Sep 17, 2025, 08:52 PM UTC

   The issue has been resolved, webhook delivery is back to normal. We continue to monitor.

3. resolved Sep 17, 2025, 09:16 PM UTC

   This incident has been resolved.

4. postmortem Sep 22, 2025, 01:05 PM UTC

   # Summary Webhook event deliveries in the EU region faced delays of up to 22 mins between 18:44 to 20:44 UTC, following an extreme level of webhook notification request timeouts. This necessitated scaling up the relevant service to mitigate. # Root Causes There was an unusual amount of webhook notification request timeouts which saturated the resources available to process webhooks. Our scale up strategy was not setup correctly to handle that particular situation. # Timeline 17:34 UTC: There began to be a large increase in request timeout to specific customer webhook endpoints 17:46 UTC: We get notified with modest abnormal webhook latency for some events 18:04 UTC: We scaled up the service’s resources, latency was stabilised 18:44 UTC: Webhook latency began to degrade significantly on all events 20:20 - 20:40 UTC: Further adjustments made to the service's autoscaling configuration 20:44 UTC: Incident fully recovered, including clearing of any queued requests # Remedies We updated our auto-scaling configuration. Follow up work planned to improve handling of abnormal request timeouts.

Read the full incident report →

Started Sep 13, 2025, 06:57 PM UTC· resolved Sep 13, 2025, 07:34 PM UTC

1. investigating Sep 13, 2025, 06:57 PM UTC

   We've identified latency updating classic dashboard results around 18:00 UTC.

2. monitoring Sep 13, 2025, 07:00 PM UTC

   A fix has been implemented. The team is now monitoring the performance and ensuring the correct propagation of all check results.

3. resolved Sep 13, 2025, 07:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 02, 2025, 10:00 PM UTC· resolved Sep 02, 2025, 11:38 PM UTC

1. identified Sep 02, 2025, 10:37 PM UTC

   Our third-party provider is performing maintenance. All Qualified Electronic Signature capture tasks are stuck in progress. Potential interruption expected up until 00:00 UTC

2. monitoring Sep 02, 2025, 11:22 PM UTC

   The maintenance is over. Service has resumed.

3. resolved Sep 02, 2025, 11:38 PM UTC

   This incident has been resolved. QES tasks are being processed as normal.

Read the full incident report →

Started Sep 01, 2025, 10:20 PM UTC· resolved Sep 02, 2025, 01:40 AM UTC

1. identified Sep 01, 2025, 10:20 PM UTC

   One of our third-party providers is experiencing technical difficulties. This impacts Turnaround times of our Identity reports.

2. identified Sep 01, 2025, 10:59 PM UTC

   One of our third-party providers continues experiencing technical difficulties. Applicants from locations outside of US and UK are facing increased Turnaround Times (TaT)

3. resolved Sep 02, 2025, 05:39 AM UTC

   This incident has been resolved

Read the full incident report →