Some portal users are experiencing access issues

Update timeline

1. identified Dec 02, 2025, 01:58 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved Dec 02, 2025, 04:39 PM UTC

   This incident has been resolved.

3. postmortem Dec 03, 2025, 02:22 PM UTC

   # **Tenant View & Access Issues for New Users \(Dec 1–2, 2025\)** **Incident Duration:** **~21 hours** — _01 Dec 2025, 19:30 UTC to 02 Dec 2025, 16:30 UTC_ ‌ **Affected Services:** Infinity Portal — _Production EU & US regions_ ‌ **Impact Scope:** Partial — _Users newly added to tenants between 30 Nov 2025 and 01 Dec 2025_ ‌ ## **Overview** Between December 1st and 2nd, newly added users in the Infinity Portal experienced **view and access issues** to their assigned tenants. This occurred due to an unintended removal of valid user-tenant associations during a database cleanup procedure. All affected data has since been fully restored, and customers have confirmed successful access. ## **Business Impact** Users added to tenants during **30 Nov 2025 – 01 Dec 2025** in the **EU and US production environments** were unable to view or access the tenants to which they were assigned. No other users or services were affected. ## **Root Cause** During an effort to correct inaccurate user access records, a cleanup process unintentionally removed valid access entries for users added between 30 Nov–01 Dec. As a result, those users temporarily lost the ability to view and access their assigned tenants. ## **Detection** The issue was detected during the deletion procedure when it became clear that legitimate entries were unintentionally removed. ## **Resolution** All missing user access records were restored using snapshot data and validated with customers. Normal access has been fully restored. ‌ ## **Timeline \(All Times in UTC\)** ‌ **30 Nov 2025 – 09:30 UTC** * A planned update was deployed as part of ongoing improvements to our user and tenant management system. **01 Dec 2025 – 15:00 UTC** * We identified irregularities in user access records that required correction to ensure accuracy and security. **01 Dec 2025 – 16:15 UTC** * A backup of the relevant databases was taken before applying corrections. **01 Dec 2025 – 16:30–19:30 UTC** * A cleanup operation was performed to remove inaccurate access records. **01 Dec 2025 – 19:30 UTC** * We discovered that the cleanup unintentionally removed valid access for users added between 30 Nov–01 Dec, causing them to lose visibility and access to their assigned tenants. **01 Dec 2025 – 19:30–21:30 UTC** * Initial restoration efforts began, and a portion of the affected access was recovered. **02 Dec 2025 – 07:30–16:30 UTC** * Full data recovery continued using database backups, restoring all missing access records. **02 Dec 2025 – 16:30 UTC** * All access issues were resolved. Customers confirmed successful restoration, and the incident was marked as complete. ‌ ## **Preventive Actions & Next Steps** * Enhanced validation workflows before performing sensitive database operations. * Enhanced our internal monitoring to ensure user access data stays accurate and consistent across our systems. * Added safeguards preventing deletion of recently created associations without confirmation. * Implementing auditing and monitoring for unexpected user-tenant association changes.