Form3 Staging experienced a minor incident on August 8, 2022 affecting Bacs - Direct Participant, lasting 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 08, 2022, 09:16 AM UTC
We have identified an issue impacting the Bacs simulator. Majority of outbound payments are in queued for delivery and no inbound payments are being received. Our engineers are currently investigating and further details will be posted as the investigation progresses
- identified Aug 08, 2022, 09:44 AM UTC
We have identified the issue and are putting in a fix in place to ensure any queued payments are processed to final state. Our Incident Response team continues to progress the recovery activity as a priority. We will provide an update as soon as more information becomes available.
- resolved Aug 08, 2022, 09:56 AM UTC
The Bacs Simulator has recovered and normal service has resumed as of 10:41 BST We will continue to monitor the service to ensure there is no ongoing impact and further details will be shared in the Post Mortem.
- postmortem Aug 09, 2022, 11:01 AM UTC
On the 8th of August, Form3 observed a degradation on the Bacs Simulator. Investigation identified that large volumes of test payments were being submitted continuously, causing unexpected load on the Bacs Simulator. This resulted in a number of test submissions becoming stuck in a non-final state, which also affected receipt of inbound test transactions. Root cause was identified, and recovery actions were taken at 10:41 BST. Once this was completed, a period of monitoring confirmed that there was no further impact to the Bacs Simulator and test transactions were being processed as expected.