Form3 Staging incident
Degradation to Staging Environment Services
Form3 Staging experienced a minor incident on February 25, 2020 affecting FPS RT - Direct and FPS RT - Indirect and 1 more component, lasting 2h 43m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 25, 2020, 08:37 AM UTC
Form3 have received an alert related a degradation to the Staging environment impacting all services. Investigations are ongoing and an update will be provided when more information becomes available.
- identified Feb 25, 2020, 09:06 AM UTC
We have identified some services have recovered and are processing as normal. Engineers are continuing to investigate degradation to the remaining services and restore normal operations as soon as possible
- resolved Feb 25, 2020, 11:20 AM UTC
All services have now recovered and following a period of monitoring Form3 can confirm normal service has resumed. Our engineers are working to put in place further mitigations against this issue and a post mortem will be posted shortly with further details
- postmortem Feb 25, 2020, 02:05 PM UTC
At 07:42 UTC we identified an issue with our services in staging. This was due to a database issue causing api calls to fail unexpectedly for all services. This issue was resolved at approximately 09:18 UTC for UK and account services while Euro services returned to operation at approximately 11:00 UTC. Following a period of monitoring and testing, Form3 confirmed all services were operational without degradation.