FinDock had 11 outages in the last 2 years totaling 608h 26m of downtime — averaging 0.5 incidents per month.
There were 11 FinDock outages since July 25, 2024 totaling 608h 26m of downtime. Each is summarised below — incident details, duration, and resolution information.
As part of our double collection protection mechanisms rolled out in last week’s hotfixes, we introduced a new bulk data monitoring export of Field History Tracking for Payment Schedule. FinDock will use this data export for more thorough monitoring of automated payment schedule behavior which enables faster, more effective troubleshooting.
Due to an ongoing incident with Salesforce Services(https://status.salesforce.com/generalmessages/10001540), some FinDock processes, particular those running on ProcessingHub, may fail or appear stuck. We are monitoring the situation closely. Currently we have not identified actions FinDock can take to help mitigate the impact.
The incident with Salesforce Services (https://status.salesforce.com/generalmessages/10001540) has seen some promising development. However, FinDock customers continue to experience disruptions, including connections issues with PSPs, blocked processes that FinDock also cannot access or resolve, processes not starting as expected, and general ProcessingHub connection instability. We are hopeful the incident will be resolved soon given the latest updates from Salesforce.
The impact of an incident with Salesforce Services (https://status.salesforce.com/generalmessages/10001540) appears to be subsiding. We are now investing if there are any actions FinDock can take to help customers recover from payment processing disruptions.
The incident with Salesforce Services (https://status.salesforce.com/generalmessages/10001540) is now resolved. Based on our investigations, there are no further ongoing payment processing disruptions. We have identified and unblocked customers that were affected and in touch with others still needing assistance. In some cases, mandate and payment schedules that where schedules to run yesterday did not successfully start on ProcessingHub. You may want to check yesterday's schedules to ensure they ran as expected. If you still run into issues ,you can contact FinDock Support.
Orgs that use FinDock permission set groups (https://docs.findock.com/setup/permission-set-groups) may be seeing permission-related errors after the January '25 production release. NOTE: Customers that use classic permission sets are not impacted! This is related to a Salesforce issue that unfortunately was not fully resolved by Salesforce. For certain orgs, when an ISV package is installed or updated, permission set groups may be reset. This removes customizations such as permission sets added from other FinDock packages and your own changes, like a muting permission set. Please follow the steps we have outlined in this new FAQ (https://docs.findock.com/faq/why-are-findock-permission-group-changes-reset) to check and restore FinDock permission set groups.
To help mitigate the permission set group issue we determined we can trigging adding permission sets from activated packages to FinDock permission set groups through another Core update. This update is being pushed right now. The update does not introduce functional changes. When the update is installed, FinDock adds permission sets from packages to the groups, and the group permissions are recalculated. This update should unblock FinDock functionality, but please note that you will still need to check and potentially re-make customizations you added to FinDock permission set groups.
Over the past few days, we have monitored the issue and restarted stalled processes that were impacted. Recently, we have not observed any further issues. Given this, we will proceed with closing the matter while continuing to work with Salesforce to ensure a resolution before our next release. If you experience any related issues, please let us know.
Some GivingPages are experiencing issues with images and fonts not rendering. Payment functionality remains unaffected, and the pages can still be used to complete transactions. FinDock has identified the issue and is working on a fix, which we intend to deploy tonight.
A fix has been implemented and all affected GivingPages are fully operational again.
**Issue**: The January '25 release included an update to the Content Security Policy \(CSP\) FinDock uses on Heroku for Giving Pages content. This update introduced stricter content rules which blocked certain page content from rendering. **Solution**: The security update blocked content hosted on the Salesforce `*.documentforce.com` domain. We’ve now added this domain to the policy whitelist to allow the content on Giving Pages.
Some UK customers that use our Bacs package are reporting an 'Undefined column" since the Jan. '25 release to production. We are actively investigating the problem and will issue a fix as soon as possible.
We have identified the problem and are now working on a fix. Once the fix is available, we will restart all blocked processes.
This incident has been resolved. A few affected customers might now run into the "Custom Encryption Key is empty" for which updates are provided via this incident: https://status.findock.com/incidents/ft89qm2802jw
For some customers, Bacs Direct Debit processes stalled on ProcessingHub due to the error `Undefined column: 7 ERROR: column "bacs_"`. This was caused by a production deployment failure that resulted in certain data tables on Heroku not being updated as expected. Re-running the related deployment actions fixed the error. We are investigating the deployment job to understand the initial failure and ensure it does not happen with future production deployments.
Some customers are reporting an empty encryption key error since the Jan. '25 release to production. We are actively investigating the problem and will issue a fix as soon as possible.
We have identified the root cause and are working on a fix to unblock all blocked processes and prevent any new blocked process.
A fix is available and currently being tested. With this fix a bug is resolved that incorrectly identified the encryption status of certain orgs. Processes blocked during the incident will be retried by FinDock. No customer action is required.
We continue to work on the fix. Ongoing monitoring and testing have revealed that some unique scenarios still cause the issue, even with the fix applied. We appreciate your patience as we work towards a comprehensive solution.
We have implemented a fix and retried all previously blocked processes affected by the customer encryption key error. As a result, all customers have been unblocked. We will continue monitoring the issue closely for any new occurrences.
We have not observed any new occurrences of the issue and have therefore closed the incident. The root cause of the incident was due to performance-related enhancements (https://docs.findock.com/release-notes/release-notes-november-24#general-performance-enhancements) resulted in FinDock incorrectly identifying the encryption status of certain orgs. This caused an empty encryption key error for these orgs that prevented processes from being run. This is now resolved.
**Issue**: Certain [performance-related enhancements](https://docs.findock.com/release-notes/release-notes-november-24#general-performance-enhancements) resulted in FinDock incorrectly identifying the encryption status of certain orgs. This caused an empty encryption key error for these orgs that prevented processes from being run. **Solution**: We identified and resolved the issue causing the error. Processes affected by the issue were automatically retried by FinDock.
We have temporarily disabled fd-eu-test-2 due to a deployment issue. Customers may need to reconnect their ProcessingHub connections in sandbox environments. This does not impact production environments.
We have resolved the issue and enabled fd-eu-test-2.
Inbound reports for Stripe charge.succeeded and payment_intent.succeeded notifications may remain in Pending status instead of being processed as expected. This issue does not impact collection, but updates to payment data in Salesforce are delayed. We are actively investigating this issue. Until we have a resolution, we recommend checking for inbound reports with Pending status and either manually retrying Guided Matching on individual reports or using automation to set the Start Guided Matching checkbox to true on the Pending reports. The next update can be expected on Monday, October 28th, at the latest.
Although we haven't seen any new occurrences of this issue, we are still investigating it. A new update will be shared by Thursday, October 31st, at the latest.
We have actively worked on this issue and developed several enhancements that are currently undergoing testing. Once all tests are successfully completed, we will deploy a patch for all customers using the Stripe for FinDock package. We will share an exact date and time for this deployment as soon as possible. If no other updates are shared beforehand, we will provide an update on Monday, November 4th.
We will be deploying an update tonight containing several enhancements. All customers using the Stripe for FinDock package will receive this patch. We will continue to monitor the situation closely to ensure that the issue is resolved.
A fix has been implemented and we are monitoring the results.
We haven't observed any new occurrences of this issue and will proceed to close the incident. We will continue to monitor the situation closely to ensure it remains resolved.
An ongoing incident at Heroku (https://status.heroku.com/incidents/2685) is giving Intermittent timeouts on GivingPages, Paylinks and Notification Gateway. We are following the situation closely and will post updates as we learn more
Heroku has provided an update that the incident has been resolved. The impact start time is 2024-10-09 13:06 UTC, and the end time is 2024-10-09 at 16:36 UTC.
Currently we are experiencing intermittent downtime for GivingPages, Paylinks and Notification Gateway. We are currently investigating the issue and provide more details later.
An ongoing incident at Heroku (https://status.heroku.com/incidents/2684) is causing the downtime. We are continuing to monitor the incident at Heroku.
Heroku has provided an update that the incident has been resolved. The impact start time is 2024-10-08 16:27 UTC, and the end time is 2024-10-08 at 22:10 UTC.
An ongoing incident at Heroku (https://status.heroku.com/incidents/2673) is preventing Giving Pages on our givingpage.org domain from rendering. This does not impact Giving Pages on custom domains. We are following the situation closely and will post updates as we learn more
We are continuing to monitor the incident at Heroku.
Heroku has provided an update that the incident has been resolved. The impact start time is 2024-07-24 18:55 UTC, and the end time is 2024-07-25 at 08:05 UTC. FinDock will continue to monitor the rendering of GivingPages on the givingpage.org domain.