Resolved - INC220338 – Central 1 UCS 43-minute Service Degradation

Update timeline

1. resolved Apr 20, 2026, 11:45 PM UTC

   Central 1 received alerts starting around 3:34 p.m. PT (6:34 p.m. ET) for UCS service degradation, impacting e-Transfers 3.4, Bill Payments, Remote Deposit Capture (RDC) Payments Services, and login services (OIDC) to Digital Banking (Forge and MemberDirect). The service degradation lasted from 3:34 to 4:17 p.m. PT (6:34 to 7:17 p.m. ET) (43-minute degradation). Services have recovered and have remained stable. Central 1 - [email protected] - 1.888.889.7878, press 1

2. postmortem Apr 29, 2026, 08:24 PM UTC

   **Postmortem: INC220338 – Universal Connectivity Services \(UCS\)** **Service Degradation** On April 20, 2026, at approximately 3:50 p.m. PT, Central 1 experienced a service degradation affecting Universal Connectivity Services \(UCS\) within the Vancouver Hosting Centre \(VAHC\). The disruption lasted approximately 43 minutes and impacted connectivity across several payment and digital banking services. During the incident window, most client traffic successfully failed over to the Toronto Hosting Centre \(TOHC\). However, a subset of clients and services that are pinned to VAHC experienced connectivity issues. Impacted services included MemberDirect and Forge login, e-Transfer 3.4, Remote Deposit Capture \(RDC\), and bill payments. Service was restored once the underlying network issue was identified and corrected. **Point of Failure:** The incident was caused by an unintended disruption to a live network cable/port during Smart Hands work performed by an Equinix technician. While completing a requested cabling change, an adjacent active connection was inadvertently disturbed, resulting in a loss of network connectivity within VAHC. Although the majority of services are designed to fail over to TOHC, applications and client connections that are statically pinned to VAHC did not automatically transition, which led to the observed service degradation for those clients. The event was detected through Dynatrace monitoring, which includes a built-in validation window to reduce false positives. The total detection and notification timeline of approximately 13 minutes was within expected operational thresholds. Post-incident review confirmed that monitoring, escalation, and response processes functioned as designed. The incident resulted in a temporary disruption to UCS-dependent services, including digital banking access and payment processing for a subset of clients. While most traffic failed over successfully, clients with dependencies tied specifically to VAHC experienced service interruption during the 43-minute window. No data loss or security concerns were identified. **Corrective Actions -** [PRB011719](https://central1.service-now.com/problem.do?sys_id=8452f90f3bd04350b117ebc964e45ad7&sysparm_record_target=problem&sysparm_record_row=1&sysparm_record_rows=1&sysparm_record_list=parent%3D682d46b23b904f10b117ebc964e45a26%5EORDERBYDESCsys_updated_on): Central 1 has completed a review with Equinix and obtained a formal incident report to validate the sequence of events and reinforce handling expectations for Smart Hands activities. Change governance practices have also been clarified to ensure stronger alignment between operational tasks and formal change management, even for activities perceived as low risk. [PTASK0010546](https://central1.service-now.com/problem_task.do?sys_id=1a391203339ccf10bbaf35bb9d5c7bd7&sysparm_record_target=problem_task&sysparm_record_row=4&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Equinix Incident Report - COMPLETED [PTASK0010543](https://central1.service-now.com/problem_task.do?sys_id=df5656c3331ccf10bbaf35bb9d5c7bda&sysparm_record_target=problem_task&sysparm_record_row=1&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Change Governance Clarity - COMPLETED * Clearer alignment between operational activities and formal change management, even when activities are perceived as low risk. [PTASK0010545](https://central1.service-now.com/problem_task.do?sys_id=37481a4b335ccf10bbaf35bb9d5c7b23&sysparm_record_target=problem_task&sysparm_record_row=3&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Investigate Port Monitoring – Due end of May 2026 * Investigate the possibility of port monitoring on high traffic ports. Is this something that should be done to better strengthen our detection process. [PTASK0010554](https://central1.service-now.com/problem_task.do?sys_id=212a42ff33dccf50bbaf35bb9d5c7b7a&sysparm_record_target=problem_task&sysparm_record_row=5&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Review failover capabilities for clients pinned to specific UCS locations – Due end of May 2026 We apologize for the disruption experienced during this incident and are taking these steps to further strengthen resiliency and change governance controls. Jason Seale | Director of Client Support Services [[email protected]](mailto:[email protected]) | 778.558.5627