INC220907 - RESOLVED: Client Centre Outage

Update timeline

1. resolved Apr 30, 2026, 08:18 PM UTC

   Please be advised that Central 1 experienced a Client Centre (https://clients.central1.com/) outage today between 12:44 to 12:57 p.m. PT (3:44 to 3:57 p.m. ET). You may have experienced issues logging into Client Centre or Central 1 applications during that time. A postmortem will be provided within the next two weeks. Central 1 - [email protected] – 1.888.889.7878, press 1

2. postmortem Jun 05, 2026, 05:56 PM UTC

   **Postmortem: INC220907 – Client Centre and Authentication Service Outage** On April 30, 2026, at approximately 12:44 p.m. PT \(3:44 p.m. ET\), Central 1 experienced a service outage impacting Client Centre and multiple authentication-dependent services. The disruption lasted approximately 23 minutes, with full service restored by 12:57 p.m. PT \(3:57 p.m. ET\). The incident occurred following a low-risk SQL database migration and subsequent decommissioning of legacy database instances. While the new database infrastructure was operating as expected, a dependency within the load balancer \(F5\) configuration caused authentication services to become unavailable. During the incident window, clients were unable to log in to Client Centre and experienced disruption accessing Central 1 applications. Impacted services included Client Centre, external ServiceNow access, PaymentStream, TeamSite CMS, Treasury Connect, and other services dependent on STS1 / ADFS authentication. This resulted in a full authentication outage for affected services during the 23-minute period. No data loss or security concerns were identified. **Point of failure:** The incident was caused by an overlooked dependency between F5 load balancer health checks and legacy SQL database instances during a database migration. F5 health probes supporting STS1 \(ADFS\) authentication services remained configured to query legacy SQL databases that had not been included in the migration scope. When those legacy databases were later placed into a read-only state or decommissioned, all F5 health checks began to fail. Service was restored after identifying the misconfigured F5 health checks and updating them to point to the new SQL database environment, allowing authentication traffic to resume normally. **Corrective Actions – PRB011728** * **PTASK0010557 – Repoint F5 health checks to new SQL databases – COMPLETED** * Updated load balancer configurations to ensure alignment with the current database environment. * **PTASK0010558 – Add F5 validation steps to DR documentation – IN PROGRESS** * Introduce formal validation steps to ensure health check dependencies are reviewed during infrastructure changes. * **PTASK0010559 – Document all F5 configurations – COMPLETED** * Improve visibility and documentation of load balancer dependencies across teams. * **PTASK0010560 – Create new validation step for database changes – COMPLETED** * Implement additional validation checkpoints within database change processes to reduce risk of missed dependencies. We apologize for the disruption experienced during this incident. Central 1 is taking these steps to strengthen change coordination, improve cross-team visibility, and ensure critical dependencies are identified and validated during infrastructure changes. Jason Seale | Director of Client Support Services [[email protected]](mailto:[email protected]) | 778.558.5627