Incident resolved: Webhub, GivingPages and PayLinks

Update timeline

1. investigating Mar 31, 2022, 09:52 AM UTC

   Currently PayLinks and GivingPages don't work in production. We have found the root cause and will implement a fix ASAP.

2. investigating Mar 31, 2022, 10:17 AM UTC

   The issue also affects WebHub. This means that notifications from Payment Service Providers can't be send to Salesforce. FinDock will fix the issue ASAP and will try to resend all failed notifications.

3. monitoring Mar 31, 2022, 11:03 AM UTC

   A fix has been deployed. Notification Gateway (WebHub), GivingPages and PayLinks are up and running again. Next steps: 1. We will actively monitor Notification Gateway, GivingPages and Paylinks 2. We will investigate if failed messages can be resend

4. resolved Mar 31, 2022, 12:34 PM UTC

   Incident resolved: No further issues have been noticed since the fix that has been deployed at 12:45 CEST. Next step: - Root cause analysis and impact analysis.

5. postmortem Mar 31, 2022, 02:31 PM UTC

   All functional applications contained in the WebHub application were non-functional due to a corruption in the shared WebHub Postgres database. This corruption was caused by an analytics pipeline pointing in the wrong direction. This caused issues with the follow components: ‌ **Giving Pages and PayLinks** Giving Pages and PayLinks were down from 11:03-12:46: * Customers were shown a 500 Server error page when trying to access a Giving page or PayLink * Giving Pages and PayLinks could not be created or adjusted from Salesforce. ‌ **Payments with and Notifications from PSPs for APIv2** Notifications, redirects and captures for specified PSPs were down from 11:03-12:22: * Notification Gateway responded with 500’s to any incoming notifications. This means that the status of existing Installments in Salesforce that were paid in this period were not updated. * For certain API v2 PSPs we rely on web pages being served from the WebHub \(Stripe, GoCardless, Redsys, PayPal, Mollie and Worldline Saferpay\) which also resulted in 500 error pages shown to the user: * Stripe, Gocardless & Redsys integrations use WebHub to redirect customers to the PSP. This means no payment could be made. * Mollie, PayPal, Saferpay integrations use WebHub to redirect customers after they have entered details on the PSP page. This means that payment was completed but the customer was not redirected to a success / thank you page. * No new WebHub accounts could be created during the downtime ‌ Failed notifications are assumed to be retried by the Payment Service Provider, which will update the Installment in Salesforce. For for Failed payment attempts through the API: * The customer was notified through an error page * An Installment was created in Salesforce with Status = New ‌ **Actions To Take** * FinDock will identify impacted customers and inform them individually * In the meantime we advise you to check Installments where the status is not “Completed” in the above time period, in case you would like to follow-up with failed payment attempts. To be notified in real-time about incidents, please subscribe to [https://status.findock.com](https://status.findock.com).