Form3 Staging experienced a minor incident on October 7, 2020 affecting Bacs - Direct Participant, lasting 3h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 07, 2020, 12:06 PM UTC
We have identified an issue impacting the Bacs simulator causing latency in payment processing. Our engineers are currently investigating and further details will be posted as the investigation progresses
- resolved Oct 07, 2020, 03:30 PM UTC
Our Bacs simulator experienced an issue with message processing latency from approximately 02:30 - 15:00 BST which has now been resolved. Queued payments have been processed and new payments are processing without issues. This was due to an issue with the service handling logic when restarting, the service has stabilised and no further impact seen. Our engineers are reviewing the handling logic for enhancement to prevent any further issues.
- postmortem Oct 08, 2020, 04:39 PM UTC
From approximately 02:30 - 15:00 BST our Bacs simulator encountered an issue where the simulator was producing higher load than usual resulting in a performance degradation in submission handling. This resulted in processing latency on all submissions until mitigations were introduced at 15:00 BST where pending submissions were processed through to a final state. A full fix has been introduced to prevent this issue recurring and we continue to monitor the simulator behaviour for further enhancements.