Form3 incident
INC154247 - Major degradation to the FPS service
Form3 experienced a critical incident on September 8, 2023 affecting Direct (MQ) and Direct (WS), lasting 1h 52m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 08, 2023, 08:24 AM UTC
Form3 are investigating a performance degradation to the FPS service. Investigations are ongoing and an update will be provided as soon as more information becomes available.
- identified Sep 08, 2023, 08:37 AM UTC
We have identified the issue, and are working on the fix. Customers may experience service degradation until the fix is in place. We will provide an update as soon as more information becomes available.
- monitoring Sep 08, 2023, 08:59 AM UTC
A fix has been implemented. Impacted customers should now see that normal processing has resumed on the FPS service. We will provide an update on resolution in due course.
- monitoring Sep 08, 2023, 09:49 AM UTC
We can confirm service has returned to normal processing and no further degradation has been seen since. We continue to monitor the service to ensure appropriate mitigation is in place 08:40 UTC
- resolved Sep 08, 2023, 10:16 AM UTC
The issue affecting FPS Instant service has been resolved as of 08:40 UTC. We have confirmed, following a period of monitoring, that the service has returned to normal processing and no further degradation has been observed. Further details will be provided in the postmortem in due course.
- postmortem Sep 11, 2023, 03:36 PM UTC
On Friday 8th September 2023, Form3 observed degradation to the Form3 - UK Instant service causing impact to both inbound and outbound payment processing between 08:09 UTC and 08:40 UTC. Recovery actions restored payment processing on the platform at 08:40 UTC. After this time additional steps were taken to remediate any queued payments. Root cause is being investigated by Form3 to identify preventative measures and put in place mitigation to prevent future issues.