Form.io incident

Performance Issues with production API's

Minor Resolved View vendor source →

Form.io experienced a minor incident on May 1, 2023 affecting Form.io API and Developer Portal, lasting 8d 13h. The incident has been resolved; the full update timeline is below.

Started
May 01, 2023, 09:47 PM UTC
Resolved
May 10, 2023, 11:46 AM UTC
Duration
8d 13h
Detected by Pingoru
May 01, 2023, 09:47 PM UTC

Affected components

Form.io APIDeveloper Portal

Update timeline

  1. investigating May 01, 2023, 09:47 PM UTC

    We are currently investigating issues with performance regarding our API's for our hosted environment @ https://portal.form.io. We recently upgraded to our latest version of 8.0.1 and there are some missing indexes used by some of our hosted customers that is making blocking database calls. We are currently adding these indexes so performance should be back up to normal momentarily.

  2. investigating May 02, 2023, 02:51 AM UTC

    We are continuing to investigate the issue with registration slowness and other queries that are being affected due to the recent upgrade.

  3. identified May 02, 2023, 03:52 AM UTC

    We have identified that the slowdown of our registrations are related to queries affecting the new version of our database. We are continuing to investigate and will update as we come out with new improvements.

  4. monitoring May 02, 2023, 01:32 PM UTC

    Registrations seem to be working, but are slow. We are continuing to investigate and are monitoring the situation and working to a resolution.

  5. monitoring May 10, 2023, 11:45 AM UTC

    We are continuing to monitor for any further issues.

  6. resolved May 10, 2023, 11:46 AM UTC

    We have mitigated most (if not all) of the outstanding performance regressions with our portal application. There are still some anomalies that we will be releasing in the coming weeks that will continue to improve performance, but as far as usability is concerned, the portal is working as expected.