Passfort experienced a minor incident on June 11, 2025 affecting 🇪🇺 EU - eu.maxsight.com, lasting 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 11, 2025, 07:49 AM UTC
We are currently seeing issues causing check delays on the platform. Our engineers are investigating this issue to identify a resolution as soon as possible.
- investigating Jun 11, 2025, 08:11 AM UTC
We are continuing to investigate the issue.
- resolved Jun 11, 2025, 08:23 AM UTC
This incident has now been resolved.
- postmortem Jul 01, 2025, 09:18 AM UTC
### Root Cause Analysis #### **Impact** Some background jobs \(inc. running checks and ingesting monitoring updates\) were unable to be processed for ~14 hours \(June 10th, 17:40UTC to June 11th 7.30UTC\). This affected BAU operations and delayed customer workflows. #### **Root Cause** The cause of this was a high number of non-critical background jobs being created and placed in one of the job queues. The lack of effective prioritisation for this job queue caused more time-critical jobs to be delayed. #### **Resolution** The non-critical jobs were manually delayed and spread out over a number of days to allow the time-critical jobs to be processed. #### **Prevention Measures** We have now deployed a number of changes to prevent this happening again; this includes better job prioritisation and also improved retry logic for certain jobs. We have also enhanced our alerting infrastructure in order to detect these types of delays earlier.