INC220235 - RESOLVED - PaymentStream Direct not accessible

Update timeline

1. investigating Apr 19, 2026, 07:04 PM UTC

   Central 1 is aware that PaymentStream Direct is not accessible and have initiated an immediate investigation. User will receive an error message that the server is unavailable after entering the 2-step token code. Keeping you informed during this incident is important to us. We’ll share another update with you by today at 2 p.m. PT (5 p.m. ET), or as soon as we have significant developments to report on. Thank you for your patience. Central 1 - [email protected] - 1.888.889.7878, press 1

2. resolved Apr 19, 2026, 07:47 PM UTC

   Service has been restarted and PaymentStream Direct is now fully accessible. Thank you for your patience during this service outage. Central 1 - [email protected] - 1.888.889.7878, press 1

3. postmortem Apr 29, 2026, 08:21 PM UTC

   **Postmortem: INC220235 – PaymentStream Direct \(PSD\) Service Outage** On Sunday, April 19, between 3:00 a.m. and 12:40 p.m. PT, PaymentStream Direct \(PSD\) experienced a service outage. During this time, the few users attempting to access PSD were unable to complete two-step verification encountering a “server is unavailable” error. The issue was resolved with a service restart, restoring the application to a healthy state. **Point of Failure:** The outage was caused by a scheduled restart of the PSD application, during which the service failed to recover properly. Although the application appeared to restart, it did not return to a functional state, as the required application port was not actively listening. This resulted in the application being unreachable despite no immediate indication of failure at the infrastructure level. The duration of the incident was extended due to gaps in monitoring for this specific unhealthy state. Automated alerting that is in place has a build in bypass of the 2SV step based upon the nature of the monitoring tool in place. This prevented the tool from detecting the outage. The existing synthetic monitoring only validated a partial authentication flow without confirming successful access into the PSD application itself. The synthetic monitoring is also being enhanced to better detect systemic failures in the 2SV workflow to raise earlier awareness. PSD, primarily used by branch staff to perform wire transfers, CRA bill payments, and AFT transactions, but also supports some Digital functions that do not require step up login. The platform has very low usage on Sundays, with ~ 1% the typical traffic of a weekday business hour. During the outage, the Payment functions were unavailable, however in the impact period it is not expected that there are branch payments originating. Some related services, including PS AFT business member transactions, were also impacted, however affected volumes are very low. A small number of credit unions did report service issues, which helped confirm the broader scope of the incident. Overall client impact was limited due to low transaction volumes during the outage window, and no data loss or security concerns were identified. **Corrective Actions -** [PRB011716](https://central1.service-now.com/problem.do?sys_id=9a5be8ae3b184710b117ebc964e45a13&sysparm_record_target=problem&sysparm_record_row=1&sysparm_record_rows=1&sysparm_record_list=parent%3Ddda01bd63358c310bbaf35bb9d5c7b97%5EORDERBYDESCsys_updated_on): Monitoring improvements are being implemented to ensure full end-to-end validation of PSD availability, including updates to synthetic monitoring to confirm successful application access rather than partial authentication checks. In parallel, application-level monitoring will be enhanced to detect and alert on port availability failures, with automated escalation through PagerDuty. [PTASK0010552](https://central1.service-now.com/problem_task.do?sys_id=cb3b1b573b540b50b117ebc964e45acf&sysparm_record_target=problem_task&sysparm_record_row=2&sysparm_record_rows=3&sysparm_record_list=problem%3D9a5be8ae3b184710b117ebc964e45a13%5EORDERBYnumber) - Improve Site24x7 Synthetic Monitoring - Completed * Re‑record Site24x7 monitor to validate full PSD access and add look for key words. [PTASK0010551](https://central1.service-now.com/problem_task.do?sys_id=d0ba9bdf3b140b50b117ebc964e45aed&sysparm_record_target=problem_task&sysparm_record_row=1&sysparm_record_rows=3&sysparm_record_list=problem%3D9a5be8ae3b184710b117ebc964e45a13%5EORDERBYnumber) - Enable Zenoss Monitoring – End of May 2026 * Add application health and port‑listening checks for PSD and ensure PagerDuty triggers automatically on failure. [PTASK0010553](https://central1.service-now.com/problem_task.do?sys_id=aaebdb133b940b50b117ebc964e45afe&sysparm_record_target=problem_task&sysparm_record_row=3&sysparm_record_rows=3&sysparm_record_list=problem%3D9a5be8ae3b184710b117ebc964e45a13%5EORDERBYnumber) - Clarify Service Ownership – End of May 2026 * Reinforce that PSD is owned by the Digital / Platform team, despite payments naming. Anthony will provide coaching to on‑call resources. We sincerely apologize for the disruption caused by this incident and are taking these steps to strengthen detection and response to prevent recurrence. Enquiries: Jason Seale | Director of Client Support Services [[email protected]](mailto:[email protected]) | 778.558.5627