Passfort Outage History

**Impact** On 04/08/2025 some customers in the EU experienced severe latency and request failures, preventing login and normal operation of the system. **Root cause** The issue occurred when an internal service responsible for Moody's screening generated more internal traffic than expected. **Resolution** ‌The service in question was temporarily blocked from making internal requests and later unblocked once it had recovered. **Prevention measures:** * The way the internal service makes requests is being improved to prevent it from overloading the system in the future

**Root Cause Analysis \(RCA\):** We identified an issue in the UAE environment where a configuration change to our infrastructure unintentionally caused some file uploads to be blocked. This was due to a modification in default behavior during the rollout. Once detected, the default behavior was promptly restored to ensure all file uploads were permitted as expected. **Resolution:** The configuration was updated to re-enable file uploads seamlessly. **Next Steps:** We are reviewing our processes to improve alerting and testing mechanisms to prevent similar issues in the future. This includes enhancing pre-deployment testing and implementing safeguards to ensure that configuration changes do not inadvertently affect functionality.

**What Happened:** Over several days, our system experienced an increase in workload that exceeded its capacity. This resulted in intermittent slowdowns in automation for multiple customers. **Resolution:** To address the issue, we scaled our system capacity, which successfully resolved the slowdowns. Automation times have now returned to normal. ‌ **Looking Ahead:** To prevent this from happening in the future, we will be implementing additional alerting on system capacity. This will help us monitor performance more closely and take proactive measures as needed.

# Root Cause Analysis ## Impact Customers may have experienced delayed monitoring updates between the 15th of June up to the 17th of July using Orbis data in Maxsight. ## Root Cause This issue occurred when the job running the sync was being killed due to a lack of memory. This meant the sync failed to complete successfully. ## Resolution After investigation, we identified memory as the issue and increased the amount available to the sync job. ## Prevention Measures * Permanently increased the amount of memory assigned to the sync job and increased the top-level limit of memory it can request.

## Incident Details Users from some institutions reported not being able to log in and could only log in using Admin account details. Incident Period: Wednesday, 9th July 9:28 AM \(BST\) - Wednesday, 9th July 2025, 12:19 PM \(BST\) ## Analysis As part of routine offboarding, some active institutions temporarily had their user accounts deactivated and API keys removed. ## Action The disabled users have had their access reinstated in active institutions. API keys have also been regenerated for these institutions. Clients have been contacted to verify that the list of active users is now correct.

### Root Cause Analysis #### **Impact** Some background jobs \(inc. running checks and ingesting monitoring updates\) were unable to be processed for ~14 hours \(June 10th, 17:40UTC to June 11th 7.30UTC\). This affected BAU operations and delayed customer workflows. #### **Root Cause** The cause of this was a high number of non-critical background jobs being created and placed in one of the job queues. The lack of effective prioritisation for this job queue caused more time-critical jobs to be delayed. #### **Resolution** The non-critical jobs were manually delayed and spread out over a number of days to allow the time-critical jobs to be processed. #### **Prevention Measures** We have now deployed a number of changes to prevent this happening again; this includes better job prioritisation and also improved retry logic for certain jobs. We have also enhanced our alerting infrastructure in order to detect these types of delays earlier.

### Root Cause Analysis #### **Impact** Some background jobs \(inc. running checks and ingesting monitoring updates\) were unable to be processed for ~3.5 hours \(June 10th, 12:30UTC to June 11th 16.00UTC\). This affected BAU operations and delayed customer workflows. #### **Root Cause** The cause of this was a high number of non-critical background jobs being created and placed in one of the job queues. The lack of effective prioritisation for this job queue caused more time-critical jobs to be delayed. #### **Resolution** The non-critical jobs were manually delayed and spread out over a number of days to allow the time-critical jobs to be processed. #### **Prevention Measures** We have now deployed a number of changes to prevent this happening again; this includes better job prioritisation and also improved retry logic for certain jobs. We have also enhanced our alerting infrastructure in order to detect these types of delays earlier.