Central 1 Outage History

**Postmortem: INC220338 – Universal Connectivity Services \(UCS\)** **Service Degradation** On April 20, 2026, at approximately 3:50 p.m. PT, Central 1 experienced a service degradation affecting Universal Connectivity Services \(UCS\) within the Vancouver Hosting Centre \(VAHC\). The disruption lasted approximately 43 minutes and impacted connectivity across several payment and digital banking services. During the incident window, most client traffic successfully failed over to the Toronto Hosting Centre \(TOHC\). However, a subset of clients and services that are pinned to VAHC experienced connectivity issues. Impacted services included MemberDirect and Forge login, e-Transfer 3.4, Remote Deposit Capture \(RDC\), and bill payments. Service was restored once the underlying network issue was identified and corrected. **Point of Failure:** The incident was caused by an unintended disruption to a live network cable/port during Smart Hands work performed by an Equinix technician. While completing a requested cabling change, an adjacent active connection was inadvertently disturbed, resulting in a loss of network connectivity within VAHC. Although the majority of services are designed to fail over to TOHC, applications and client connections that are statically pinned to VAHC did not automatically transition, which led to the observed service degradation for those clients. The event was detected through Dynatrace monitoring, which includes a built-in validation window to reduce false positives. The total detection and notification timeline of approximately 13 minutes was within expected operational thresholds. Post-incident review confirmed that monitoring, escalation, and response processes functioned as designed. The incident resulted in a temporary disruption to UCS-dependent services, including digital banking access and payment processing for a subset of clients. While most traffic failed over successfully, clients with dependencies tied specifically to VAHC experienced service interruption during the 43-minute window. No data loss or security concerns were identified. **Corrective Actions -** [PRB011719](https://central1.service-now.com/problem.do?sys_id=8452f90f3bd04350b117ebc964e45ad7&sysparm_record_target=problem&sysparm_record_row=1&sysparm_record_rows=1&sysparm_record_list=parent%3D682d46b23b904f10b117ebc964e45a26%5EORDERBYDESCsys_updated_on): Central 1 has completed a review with Equinix and obtained a formal incident report to validate the sequence of events and reinforce handling expectations for Smart Hands activities. Change governance practices have also been clarified to ensure stronger alignment between operational tasks and formal change management, even for activities perceived as low risk. [PTASK0010546](https://central1.service-now.com/problem_task.do?sys_id=1a391203339ccf10bbaf35bb9d5c7bd7&sysparm_record_target=problem_task&sysparm_record_row=4&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Equinix Incident Report - COMPLETED [PTASK0010543](https://central1.service-now.com/problem_task.do?sys_id=df5656c3331ccf10bbaf35bb9d5c7bda&sysparm_record_target=problem_task&sysparm_record_row=1&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Change Governance Clarity - COMPLETED * Clearer alignment between operational activities and formal change management, even when activities are perceived as low risk. [PTASK0010545](https://central1.service-now.com/problem_task.do?sys_id=37481a4b335ccf10bbaf35bb9d5c7b23&sysparm_record_target=problem_task&sysparm_record_row=3&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Investigate Port Monitoring – Due end of May 2026 * Investigate the possibility of port monitoring on high traffic ports. Is this something that should be done to better strengthen our detection process. [PTASK0010554](https://central1.service-now.com/problem_task.do?sys_id=212a42ff33dccf50bbaf35bb9d5c7b7a&sysparm_record_target=problem_task&sysparm_record_row=5&sysparm_record_rows=5&sysparm_record_list=problem%3D8452f90f3bd04350b117ebc964e45ad7%5EORDERBYnumber) - Review failover capabilities for clients pinned to specific UCS locations – Due end of May 2026 We apologize for the disruption experienced during this incident and are taking these steps to further strengthen resiliency and change governance controls. Jason Seale | Director of Client Support Services [[email protected]](mailto:[email protected]) | 778.558.5627

**Postmortem: INC220235 – PaymentStream Direct \(PSD\) Service Outage** On Sunday, April 19, between 3:00 a.m. and 12:40 p.m. PT, PaymentStream Direct \(PSD\) experienced a service outage. During this time, the few users attempting to access PSD were unable to complete two-step verification encountering a “server is unavailable” error. The issue was resolved with a service restart, restoring the application to a healthy state. **Point of Failure:** The outage was caused by a scheduled restart of the PSD application, during which the service failed to recover properly. Although the application appeared to restart, it did not return to a functional state, as the required application port was not actively listening. This resulted in the application being unreachable despite no immediate indication of failure at the infrastructure level. The duration of the incident was extended due to gaps in monitoring for this specific unhealthy state. Automated alerting that is in place has a build in bypass of the 2SV step based upon the nature of the monitoring tool in place. This prevented the tool from detecting the outage. The existing synthetic monitoring only validated a partial authentication flow without confirming successful access into the PSD application itself. The synthetic monitoring is also being enhanced to better detect systemic failures in the 2SV workflow to raise earlier awareness. PSD, primarily used by branch staff to perform wire transfers, CRA bill payments, and AFT transactions, but also supports some Digital functions that do not require step up login. The platform has very low usage on Sundays, with ~ 1% the typical traffic of a weekday business hour. During the outage, the Payment functions were unavailable, however in the impact period it is not expected that there are branch payments originating. Some related services, including PS AFT business member transactions, were also impacted, however affected volumes are very low. A small number of credit unions did report service issues, which helped confirm the broader scope of the incident. Overall client impact was limited due to low transaction volumes during the outage window, and no data loss or security concerns were identified. **Corrective Actions -** [PRB011716](https://central1.service-now.com/problem.do?sys_id=9a5be8ae3b184710b117ebc964e45a13&sysparm_record_target=problem&sysparm_record_row=1&sysparm_record_rows=1&sysparm_record_list=parent%3Ddda01bd63358c310bbaf35bb9d5c7b97%5EORDERBYDESCsys_updated_on): Monitoring improvements are being implemented to ensure full end-to-end validation of PSD availability, including updates to synthetic monitoring to confirm successful application access rather than partial authentication checks. In parallel, application-level monitoring will be enhanced to detect and alert on port availability failures, with automated escalation through PagerDuty. [PTASK0010552](https://central1.service-now.com/problem_task.do?sys_id=cb3b1b573b540b50b117ebc964e45acf&sysparm_record_target=problem_task&sysparm_record_row=2&sysparm_record_rows=3&sysparm_record_list=problem%3D9a5be8ae3b184710b117ebc964e45a13%5EORDERBYnumber) - Improve Site24x7 Synthetic Monitoring - Completed * Re‑record Site24x7 monitor to validate full PSD access and add look for key words. [PTASK0010551](https://central1.service-now.com/problem_task.do?sys_id=d0ba9bdf3b140b50b117ebc964e45aed&sysparm_record_target=problem_task&sysparm_record_row=1&sysparm_record_rows=3&sysparm_record_list=problem%3D9a5be8ae3b184710b117ebc964e45a13%5EORDERBYnumber) - Enable Zenoss Monitoring – End of May 2026 * Add application health and port‑listening checks for PSD and ensure PagerDuty triggers automatically on failure. [PTASK0010553](https://central1.service-now.com/problem_task.do?sys_id=aaebdb133b940b50b117ebc964e45afe&sysparm_record_target=problem_task&sysparm_record_row=3&sysparm_record_rows=3&sysparm_record_list=problem%3D9a5be8ae3b184710b117ebc964e45a13%5EORDERBYnumber) - Clarify Service Ownership – End of May 2026 * Reinforce that PSD is owned by the Digital / Platform team, despite payments naming. Anthony will provide coaching to on‑call resources. We sincerely apologize for the disruption caused by this incident and are taking these steps to strengthen detection and response to prevent recurrence. Enquiries: Jason Seale | Director of Client Support Services [[email protected]](mailto:[email protected]) | 778.558.5627

Postmortem: INC219954 – Intermittent Two‑Step Verification \(2SV\) Login Errors On Monday, April 14 2026, beginning at approximately 12:30 a.m. PT, some credit union customers experienced intermittent login failures when accessing Retail and Small Business Online Banking. The issue persisted until 10:10 a.m. PT, when full service was restored. Customers affected by the issue may have been unable to complete logins requiring Two‑Step Verification \(2SV\). Customers using biometric login or unchallenged 2SV flows were not impacted. Point of Failure: One of the newly added MD Authentication \(MDAuth\) servers entered a degraded state following its routine nightly restart. Although the server appeared operational, it was unable to reliably process 2SV authentication requests, resulting in intermittent login failures for users routed to that server. Under normal circumstances, Dynatrace monitoring alerts the team when an authentication server enters a degraded state. However, this newly added server was configured with an incorrect monitoring mode, preventing expected alerts from triggering. As a result, the server remained in rotation longer than expected while serving failed authentication requests. Resolution: The affected MDAuth server was temporarily removed from the load balancer, immediately stabilizing 2SV login functionality. No further issues were observed following its removal. Corrective Actions: Server Removed from Load Balancer – COMPLETED The affected authentication server was suspended from traffic to restore service stability. Monitoring Configuration Corrected – COMPLETED Dynatrace monitoring has been updated to ensure the server is tracked under the correct monitoring mode and generates alerts for this failure condition. Post‑Deployment Validation Review – COMPLETED Monitoring configuration checks have been reinforced for newly added authentication servers to ensure alerting is active prior to placing them into production rotation. We apologize for the disruption this caused to your operations and your customers. If you have any questions or would like additional detail, please contact Digital Banking Support. Digital Banking Support [digitalbanking\[email protected]](mailto:[email protected]) 1\.888.889.7878 \(press 2\)

**INC219290 Postmortem: Wire Transfer Service Disruption** On April 2, 2026, between 5:25 a.m. PT and 10:21 a.m. PT \(8:25 a.m. to 1:21 p.m. ET\), Central 1 experienced a service disruption impacting incoming and outgoing wire transfers. During this time, wire creation through PaymentStream Direct remained available; however, transmission, processing, and transaction inquiry functions were unavailable or significantly delayed due to an issue within a third-party service provider. The disruption was caused by failures within Fiserv’s Enterprise Payments Platform \(EPP\), which supports wire processing and regulatory screening. As a result, wires could not be transmitted or processed, leading to delays in time-sensitive payments and increased client escalations. Service was fully restored after the vendor applied corrective actions, and normal processing resumed. **Point of Failure:** The incident originated within Fiserv’s infrastructure across multiple layers. An automated process within Fiserv’s environment incorrectly modified hostname configurations on IBM MQ gateway servers, preventing MQ services from starting and halting wire message processing. In parallel, the required database configuration settings were not consistently applied in the older environment, leading to degraded performance, timeouts, and instability in the Enterprise Payments Platform. Together, these issues led to a complete interruption of wire processing services. **Corrective Actions** Fiserv implemented emergency changes to restore MQ services and correct database configuration settings, stabilizing the platform and resuming normal wire processing. As part of their follow-up actions, Fiserv is standardizing database configurations across all environments, reinforcing installation and configuration practices with their teams, and introducing validation controls to ensure required settings are consistently applied. At Central 1, we are enhancing monitoring to improve early detection of third-party processing failures and reviewing our vendor support and escalation processes to ensure faster resolution. We are also assessing our internal incident handling procedures to strengthen coordination and response effectiveness during third-party outages. * PTASK0010523 - FISERV Incident and Root Cause Report – COMPLETED * PTASK0010524 - Improved Wires Monitoring - COMPLETED * PTASK0010526 – Assess Document C1 Incident Handling Procedures – Due end of April 2026 * PTASK0010525 - C1 Vendor support Process Review - Due end of Q2 2026 We sincerely apologize for the disruption and the impact this incident had on your operations, particularly for time-sensitive wire transactions, and we appreciate your patience while services were restored.

**Postmortem: INC219263 – UCS Service Degradation Impacting Payments and Digital Banking** On April 1, 2026, between 4:17 p.m. and 4:34 p.m. PT \(7:17 p.m. to 7:34 p.m. ET\), Central 1 experienced a service degradation affecting multiple payment and authentication services. During this 17-minute window, clients may have experienced intermittent issues with Interac e-Transfers \(3.4/Iso8583\), Bill Payments, Remote Deposit Capture \(RDC\), and login services supporting the Forge and MemberDirect Digital Banking platforms. Service performance degraded, resulting in 50% traffic failing in the impact window. Service self-recovered at 4:34 p.m. PT \(7:34 p.m. ET\) , with all services remaining stable thereafter. We recognize the importance of these services to your operations and sincerely apologize for the disruption experienced during this time. **Point of Failure**: The degradation coincided with an unauthorized infrastructure event involving Central 1’s virtual firewall environment in Microsoft Azure. A hot-plug activity \(adding or removing a component or device from a server while it is active\) performed by our vendor, Palo Alto Networks, triggered an unexpected firewall reboot during business hours. This resulted in a brief interruption to network traffic, impacting upstream services. The issue was resolved by the vendor without requiring intervention from Central 1, and services recovered immediately following the event. **PRB011705 - Corrective Actions:** * PTASK0010529 – Vendor Patching Maintenance Enhancement - COMPLETED Central 1 has formally escalated this incident with Palo Alto Networks to reinforce adherence to agreed maintenance windows and change management expectations. We are working with the vendor to ensure that any future infrastructure activities are either conducted within approved maintenance windows or communicated in advance to prevent unintended service impact. Internally, we have opened a high-priority problem record \(PRB011705\) to further assess monitoring, vendor coordination, and response readiness, ensuring improved visibility and faster validation of root cause in similar scenarios. We recognize how important these services are to your operations and apologize for the disruption, truly appreciating your patience during this event. Jason Seale | Director of Client Support Services | [[email protected]](mailto:[email protected]) 778.558.5627

**INC219102 – Error When Accessing the Client Centre** On March 30, 2026, between 3:00 a.m. PT \(6:00 a.m. ET\) and 4:15 a.m. PT \(7:15 a.m. ET\), Central 1 customers experienced intermittent access issues \(approximately 50%\) when accessing Client Centre and related services and applications. The issue was caused by how login requests were being directed within the system. While traffic is normally spread across available servers, some login attempts were sent to servers that were not available. Because the system tried to keep users on the same path for consistency, repeated login attempts continued to be directed to the same unavailable server. **Point of Failure** This incident was triggered by a routine system change made on Sunday, March 29, 2026. A background process responsible for managing system capacity did not complete successfully. This did not have an immediate impact but became apparent once normal business traffic increased, at which point the system reached its limits. As a result of this condition, some login requests could not be properly processed, leading to login failures for approximately half of the users. In some cases, affected users continued to experience the issue on repeated attempts until they refreshed their session \(for example, by closing and reopening their browser\). To restore service, the two impacted login servers were removed from service, allowing traffic to be handled by the remaining healthy servers and resolving the issue. **PRB011701 – Actions & Lessons Learned** Central 1 has reinforced the requirement for all teams to create change tickets and adhere to the established change management process. * **PTASK0010527** – Monitoring did not alert on disk usage prior to the server reaching full capacity – _Complete_ * **PTASK0010532** – Investigate enhancing ADFS probes beyond basic HTTP checks – _Target: Q2 2026_ We apologize for the disruption this caused to your operations and your members. If you have any questions or require additional details, please contact Support. Central 1 – Support | [[email protected]](mailto:[email protected]) | 1.888.889.7878 \(press 1\)

Postmortem: INC217878 – Intermittent Two‑Step Verification \(2SV\) SMS Issues On March 5th 2026, beginning at approximately 12:39 a.m. PT \(3:39 a.m. ET\), some customers experienced intermittent issues when attempting to complete Two‑Step Verification \(2SV\) via SMS. Approximately 25% of SMS‑challenged logins and transactional step‑ups were impacted during this period. Full service was restored by 8:31 a.m. PT \(11:31 a.m. ET\). Customers affected by the issue may have received error messages such as “code cannot be delivered” or “invalid code.” In most cases, retrying the action resulted in a successful authentication. Other authentication methods were unaffected. Point of Failure: One of the MD authentication servers entered a degraded state following a restart. Although the server appeared online, it was unable to reliably process SMS‑based 2SV requests, resulting in intermittent delivery and validation failures for users routed to that server. The error observed in system logs differed from previously encountered authentication server failures and was not immediately detected by existing alerting. Because the server was not fully offline, it remained in rotation longer than expected, impacting approximately one‑quarter of SMS‑challenged authentication attempts. The affected authentication server was removed from the service pool, immediately stabilizing 2SV SMS functionality. Corrective Actions: Affected Server Removed – COMPLETED The MD authentication server in a degraded state was removed from the authentication pool, restoring normal service. Enhanced Alerting for New Error Pattern – COMPLETED Monitoring and alerting have been updated to detect this specific log error earlier, enabling faster identification and response in similar scenarios. Operational Review – COMPLETED The incident was reviewed against existing restart and recovery procedures to ensure alignment and identify any additional improvements. We apologize for the inconvenience this caused to your operations and your customers. If you have any questions or would like additional detail, please contact Digital Banking Support. Digital Banking Support [digitalbanking\[email protected]](mailto:[email protected]) 1\.888.889.7878 \(press 2\)

**Postmortem: INC217374 – e-Transfer 3.5/ISO20022 Outage** On Monday, February 23, 2026, Central 1 experienced a service disruption to e-Transfer version 3.5 \(ISO 20022\) between 7:35 pm PT \(10:35 pm ET\) and 8:47 pm PT \(11:47 pm ET\), which affected both send and receive transactions. When service was restored, Auto-deposits received during that time were automatically retried, requiring no further action. There was no impact to 3.4/ISO8583 transactions. **Point of Failure:** The outage was caused by a failure within the messaging infrastructure supporting Interac e-Transfer 3.5, specifically related to the Amazon MQ \(ActiveMQ\) PSA Trace broker reaching full storage capacity. This occurred after two trace consumer instances experienced a sustained, unexplained degradation in network throughput beginning on February 19, significantly reducing their ability to process trace messages. As a result, messages accumulated in both the primary trace queue and dead-letter queue, preventing the system from clearing processed data and ultimately exhausting available storage. Once the broker reached capacity, built-in flow control mechanisms halted all message publishing across the platform, which in turn blocked payment processing and led to downstream application failures, including loss of connectivity within the EMT service. Although the issue originated in a non-functional trace processing component, it had a cascading impact on core payment flows. Service was restored by clearing the accumulated backlog and restarting affected services, while AWS continues to investigate the underlying cause of the EC2 network degradation. **Corrective Actions \(PRB011685\):** * Add monitoring & alerts for PSA trace broker – Complete * Evaluate disabling PSA trace queues \(partially or fully\): Q2 – 2026 * AWS Investigation – Ongoing by vendor We apologize for the delay and appreciate your patience while Central 1 works internally to improve system functionality. Central 1 - [[email protected]](mailto:[email protected]) - 1.888.889.7878