User Login Access Issue

Update timeline

1. investigating Jul 02, 2026, 03:22 PM UTC

   We are aware of an issue affecting user login for certain accounts. Impacted users may experience a spinning loading screen or an error when attempting to authenticate. Our team has confirmed the issue, and it is currently under active investigation. Impact: Affected users are unable to access the system. No workaround is currently available. Next Update: We will provide an update once the root cause has been identified or a fix is in progress.

2. investigating Jul 02, 2026, 04:32 PM UTC

   Update: We have determined that this issue is intermittently affecting a limited subset of customers. Based on our current assessment, we are downgrading the impact level from an outage to a service degradation. Our engineering teams continue to actively investigate the issue and are working to identify the root cause. We will provide additional updates as more information becomes available. Impact: Affected users are unable to access the system. No workaround is currently available. Next Update: As progress is made in the investigation

3. monitoring Jul 02, 2026, 04:46 PM UTC

   Update: The issue has been resolved, and service has been restored for affected customers. Our teams have implemented corrective actions and validated the recovery of impacted services. We are continuing to closely monitor system performance and stability to ensure services remain healthy and operating as expected. Our investigation into the underlying cause remains ongoing. A detailed Root Cause Analysis (RCA), including corrective and preventive actions, will be made available within 7–10 business days. Current Status: Resolved Customer Impact: Mitigated Next Update: We will provide the RCA within 7–10 business days.

4. resolved Jul 02, 2026, 07:07 PM UTC

   Update: The issue remains resolved, and we have not observed any new issues since service restoration. System performance and stability continue to operate within expected parameters. Our teams will continue monitoring as a precaution, but no further customer impact has been identified. Current Status: Resolved Customer Impact: None observed Next Update: A Root Cause Analysis (RCA) will be provided within 7–10 business days.

5. postmortem Jul 14, 2026, 01:27 PM UTC

   **Incident Summary** On July 2, 2026, between 08:04 PT and 09:30 PT, users may have experienced significant slowness or "hanging" when attempting to log in or save records within the application. The issue was caused by a degradation in database performance that impacted several core services. **Detection** The issue was identified at 08:04 PT following internal reports of application unresponsiveness. Engineering teams immediately initiated an incident response bridge to diagnose the cause of the service degradation. **Root Cause** The degradation was caused by a background database query used for indexing configuration fields. This query lacked specific filters, causing it to scan and sort an excessive amount of data for certain accounts. This activity affected the database’s processing power \(CPU\), leading to a backlog that delayed other essential tasks, such as login requests and data saves. **Resolution** To resolve the issue, Engineering transitioned operations to a fresh database instance to immediately restore processing capacity. Once the system was stable, a new database index was implemented to optimize the problematic query, allowing it to run efficiently without impacting system resources. Service returned to normal levels by 09:30 PT. **Preventative Measures** To prevent this from occurring again, the following measures are being implemented: * **Code Optimization:** The background indexing logic is being updated to include stricter data filters, reducing the load on the database. * **Improved Monitoring:** We are adding new automated alerts to detect high database resource consumption and long-running queries before they impact users. * **Infrastructure Review:** The new database optimizations are being deployed across all environments to ensure consistent performance.