Passfort experienced a minor incident on June 10, 2025 affecting 🇪🇺 EU - eu.maxsight.com, lasting 1h 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 10, 2025, 02:58 PM UTC
We are currently seeing issues causing check delays on the platform. Our engineers are investigating this issue to identify a resolution as soon as possible.
- identified Jun 10, 2025, 04:22 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Jun 10, 2025, 04:54 PM UTC
This incident has been resolved.
- postmortem Jul 01, 2025, 09:15 AM UTC
### Root Cause Analysis #### **Impact** Some background jobs \(inc. running checks and ingesting monitoring updates\) were unable to be processed for ~3.5 hours \(June 10th, 12:30UTC to June 11th 16.00UTC\). This affected BAU operations and delayed customer workflows. #### **Root Cause** The cause of this was a high number of non-critical background jobs being created and placed in one of the job queues. The lack of effective prioritisation for this job queue caused more time-critical jobs to be delayed. #### **Resolution** The non-critical jobs were manually delayed and spread out over a number of days to allow the time-critical jobs to be processed. #### **Prevention Measures** We have now deployed a number of changes to prevent this happening again; this includes better job prioritisation and also improved retry logic for certain jobs. We have also enhanced our alerting infrastructure in order to detect these types of delays earlier.