FinDock incident

ProcessingHub production down for some customers

Major Resolved View vendor source →

FinDock experienced a major incident on November 28, 2022 affecting File processing (ProcessingHub), lasting 4h 54m. The incident has been resolved; the full update timeline is below.

Started
Nov 28, 2022, 10:24 AM UTC
Resolved
Nov 28, 2022, 03:19 PM UTC
Duration
4h 54m
Detected by Pingoru
Nov 28, 2022, 10:24 AM UTC

Affected components

File processing (ProcessingHub)

Update timeline

  1. investigating Nov 28, 2022, 10:24 AM UTC

    Some customers are experiencing issues on ProcessingHub, preventing them to run Payment Schedules, Mandate Schedules and uploading files. We are investigating the cause impacted customers and will provide updates on status.findock.com

  2. identified Nov 28, 2022, 10:51 AM UTC

    Update: The issue has been caused by a change in our Nov. '22 release to production. We have created the fix and are now finalising the regression tests. Once finished the fix will be pushed to all production environments.

  3. identified Nov 28, 2022, 01:21 PM UTC

    Update: The fix has been deployed and all blocked and stuck processes have been restarted. We'll keep on monitoring the fix for the coming hours.

  4. monitoring Nov 28, 2022, 01:23 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Nov 28, 2022, 03:19 PM UTC

    All stuck and blocked processes have been processed. The issue has been introduced by this enhancement (https://docs.findock.com/release-notes-november-22#file-operation-errors-in-sandboxes-with-enhanced-domains) of the FinDock Nov. '22 release, which is pushed to production on November 27th. With this enhancement, FinDock introduced checking the instance URL based on the Salesforce identity URL. In production this check caused issues for some of our customers. This has now been resolved.