Harness Outage History

## **Summary** On **February 4, 2026**, SEI experienced a temporary outage that affected SEI Insights. SEI Customers \(US production clusters\) were unable to see Insights data during this period. Customer data remained safe and was not impacted. ## **Root Cause** An automated system update caused our main database service to stop working correctly. When it restarted, it did not recover as expected, which led to temporary service errors. ## **Impact** * **Customer impact:** SEI Insights did not display data * **Duration:** Approximately **9:33 AM to 2:37 PM PST** * **Scope:** SEI Insights Only * **Data safety:** No customer data was lost or damaged ## **Mitigation and Permanent Fix** To restore service, we created and switched to a new, more resilient database setup. This allowed SEI Insights to return to normal operation. ## **Next Steps** * Keeping a backup system ready to reduce downtime if a similar issue occurs * Closely monitoring the new setup to quickly detect and address issues

## **Summary** On Feb 3rd, 2026, customers using the Feature Flags module in the production environment \(Prod2\) observed delays in seeing their updates reflected in the user interface. The issue was caused by lag in a database read replica, which resulted in stale data being served for read operations. The issue was identified, mitigated, and fully resolved. ## **Impact** During the incident window: * Updates made to Feature Flags Classic in Prod2 were not immediately reflected in the UI. * Read operations returned stale data due to replication lag between the primary database and a read replica. * All Feature Flags Classic customers in Prod2 were affected. There was **no data loss**, and write operations continued to be processed successfully. The impact was limited to delayed visibility of updates and temporary confusion regarding the status of recent changes. Overall service availability was slightly degraded during the incident. ## **Root Cause** Feature Flags Classic relies on a primary database for write operations and read replicas for read operations. During the incident, a long-running database query caused a read replica to fall significantly behind the primary database. ‌ As a result, while customer updates were successfully written to the primary database, reads served from the lagging replica returned outdated data. Because replication lag alerts were not enabled at the time, the issue was not detected immediately through automated monitoring. ## **Mitigation** As immediate mitigation steps: * The long-running query on the affected read replica was terminated. * The replica was restarted, allowing it to catch up with the primary database and resume normal operation. These actions restored data consistency between the primary and replica databases and resolved the customer-facing impact. ## **Action Items** To reduce the risk of recurrence and improve detection, the following actions are being implemented: * Enable proactive monitoring and alerting for database replication lag with defined thresholds. * Configure query timeouts to prevent long-running queries from impacting database replicas. * Establish clearer operational guidelines for executing resource-intensive queries. * Review and periodically validate database alerting configurations to ensure early detection of similar issues.

## **Summary** On February 3, 2026, dashboards in the Prod4 environment experienced data-fetch failures following the deployment . The issue was caused by an unintended configuration change prevented dashboard metrics from loading.The issue was identified and mitigated the same day through configuration correction and service redeployment. ## **Impact** * Dashboard metrics failed to load in the Prod4 environment. * Underlying system functionality remained operational. * No data loss occurred. * The issue was limited to dashboard data visibility. ## **Root Cause** A recent update to a shared configuration chart was introduced to support an unrelated migration initiative. This change resulted in authentication failures, preventing services from connecting to the database and loading dashboard data ## **Mitigation** The issue was mitigated through the following steps: * Reverted the shared configuration to restore normal functionality. * Applied an emergency configuration fix to explicitly set the correct values. * Rebuilt and redeployed all affected services across relevant versions. ## **Action Items** To prevent recurrence, the following improvements are being implemented: * Enhance version control mechanisms to prevent automatic adoption of shared configuration changes without explicit service-level upgrades. * Improve configuration precedence logic to ensure safer fallback mechanisms. * Enhance monitoring and validation for all dependency connectivity. * Strengthen regression validation processes for shared configuration changes.