Form3 Staging incident

Degradation to Bacs Simulator

Resolved after 3 hours 23 minutes, affecting Bacs - Direct Participant, on October 7, 2020.

postmortem Oct 08, 2020, 04:39 PM UTC

From approximately 02:30 - 15:00 BST our Bacs simulator encountered an issue where the simulator was producing higher load than usual resulting in a performance degradation in submission handling. This resulted in processing latency on all submissions until mitigations were introduced at 15:00 BST where pending submissions were processed through to a final state. A full fix has been introduced to prevent this issue recurring and we continue to monitor the simulator behaviour for further enhancements.

Started
Oct 07, 2020, 12:06 PM UTC
Resolved
Oct 07, 2020, 03:30 PM UTC
Duration
3h 23m
Detected by Pingoru
Oct 07, 2020, 12:06 PM UTC

Affected components

Bacs - Direct Participant

Update timeline

  1. investigating Oct 07, 2020, 12:06 PM UTC

    We have identified an issue impacting the Bacs simulator causing latency in payment processing. Our engineers are currently investigating and further details will be posted as the investigation progresses

  2. resolved Oct 07, 2020, 03:30 PM UTC

    Our Bacs simulator experienced an issue with message processing latency from approximately 02:30 - 15:00 BST which has now been resolved. Queued payments have been processed and new payments are processing without issues. This was due to an issue with the service handling logic when restarting, the service has stabilised and no further impact seen. Our engineers are reviewing the handling logic for enhancement to prevent any further issues.

  3. postmortem Oct 08, 2020, 04:39 PM UTC

    From approximately 02:30 - 15:00 BST our Bacs simulator encountered an issue where the simulator was producing higher load than usual resulting in a performance degradation in submission handling. This resulted in processing latency on all submissions until mitigations were introduced at 15:00 BST where pending submissions were processed through to a final state. A full fix has been introduced to prevent this issue recurring and we continue to monitor the simulator behaviour for further enhancements.

Your product runs on more than Form3 Staging

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →