Form3 incident
#125221 - Major Outage to UK instant, UK Bacs, SEPA and SEPA Instant services
Form3 experienced a critical incident on June 28, 2023 affecting Credit Transfer - Indirect and Indirect and 1 more component, lasting 1h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 28, 2023, 09:16 AM UTC
Form3 are investigating a performance degradation to the UK instant and SEPA Instant services. Investigations are ongoing and an update will be provided as soon as more information becomes available.
- investigating Jun 28, 2023, 09:20 AM UTC
We are continuing to investigate this issue.
- investigating Jun 28, 2023, 09:22 AM UTC
Form3 are investigating an outage to the UK instant, UK Bacs, Confirmation of Payee, SEPA and SEPA Instant services. Admissions and Submissions are unavailable. Investigations are ongoing and an update will be provided as soon as more information becomes available.
- monitoring Jun 28, 2023, 09:35 AM UTC
A fix has been implemented and we are in a period of monitoring. Impacted customers should now see that normal processing has resumed on all services. We will provide an update on resolution in due course.
- resolved Jun 28, 2023, 10:19 AM UTC
The issue affecting UK Instant, Confirmation of Payee, SEPA and SEPA Instant services has been resolved as of 09:31 UTC. We have confirmed, following a period of monitoring, that the service has returned to normal processing and no further degradation has been observed. We have further confirmed UK Bacs was not impacted by this outage as previously reported. Further details will be provided in the postmortem in due course.
- postmortem Jun 29, 2023, 07:03 PM UTC
On 28 June at 09:50 BST, Form3 observed a major outage on multiple services causing a delay in inbound and outbound payment processing. The following services were impacted: UK Instant, Form3 – SEPA Instant, Form3 – SEPA and Form3 – Confirmation of Payee Recovery actions restored payment processing on the platform at 10:31 BST. After this time additional steps were taken to remediate any queued payments. Root cause has been isolated to one of our cloud providers failing to upload an updated certificate. Form3 are implementing preventative measures to avoid the risk of reoccurrence.