FinDock incident
Bacs processes stalled due to error 'Undefined column: 7 ERROR: column "bacs_'
FinDock experienced a major incident on January 27, 2025 affecting File processing (ProcessingHub), lasting 5h 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 27, 2025, 09:35 AM UTC
Some UK customers that use our Bacs package are reporting an 'Undefined column" since the Jan. '25 release to production. We are actively investigating the problem and will issue a fix as soon as possible.
- identified Jan 27, 2025, 11:26 AM UTC
We have identified the problem and are now working on a fix. Once the fix is available, we will restart all blocked processes.
- resolved Jan 27, 2025, 03:31 PM UTC
This incident has been resolved. A few affected customers might now run into the "Custom Encryption Key is empty" for which updates are provided via this incident: https://status.findock.com/incidents/ft89qm2802jw
- postmortem Jan 28, 2025, 09:56 AM UTC
For some customers, Bacs Direct Debit processes stalled on ProcessingHub due to the error `Undefined column: 7 ERROR: column "bacs_"`. This was caused by a production deployment failure that resulted in certain data tables on Heroku not being updated as expected. Re-running the related deployment actions fixed the error. We are investigating the deployment job to understand the initial failure and ensure it does not happen with future production deployments.