FinDock incident
Blocked processes due to error on ProcessingHub 'Custom encryption key is empty'
FinDock experienced a notice incident on January 27, 2025 affecting File processing (ProcessingHub), lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 27, 2025, 09:19 AM UTC
Some customers are reporting an empty encryption key error since the Jan. '25 release to production. We are actively investigating the problem and will issue a fix as soon as possible.
- identified Jan 27, 2025, 10:27 AM UTC
We have identified the root cause and are working on a fix to unblock all blocked processes and prevent any new blocked process.
- identified Jan 27, 2025, 11:05 AM UTC
A fix is available and currently being tested. With this fix a bug is resolved that incorrectly identified the encryption status of certain orgs. Processes blocked during the incident will be retried by FinDock. No customer action is required.
- identified Jan 27, 2025, 02:03 PM UTC
We continue to work on the fix. Ongoing monitoring and testing have revealed that some unique scenarios still cause the issue, even with the fix applied. We appreciate your patience as we work towards a comprehensive solution.
- monitoring Jan 27, 2025, 04:03 PM UTC
We have implemented a fix and retried all previously blocked processes affected by the customer encryption key error. As a result, all customers have been unblocked. We will continue monitoring the issue closely for any new occurrences.
- resolved Jan 28, 2025, 09:54 AM UTC
We have not observed any new occurrences of the issue and have therefore closed the incident. The root cause of the incident was due to performance-related enhancements (https://docs.findock.com/release-notes/release-notes-november-24#general-performance-enhancements) resulted in FinDock incorrectly identifying the encryption status of certain orgs. This caused an empty encryption key error for these orgs that prevented processes from being run. This is now resolved.
- postmortem Jan 28, 2025, 09:58 AM UTC
**Issue**: Certain [performance-related enhancements](https://docs.findock.com/release-notes/release-notes-november-24#general-performance-enhancements) resulted in FinDock incorrectly identifying the encryption status of certain orgs. This caused an empty encryption key error for these orgs that prevented processes from being run. **Solution**: We identified and resolved the issue causing the error. Processes affected by the issue were automatically retried by FinDock.