Form.io experienced a critical incident on August 29, 2023, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Aug 30, 2023, 02:44 PM UTC
At approximately 3:10PM, we experienced an API outage that lasted for approximately 10 minutes on our Hosted platform at https://portal.form.io and the API traffic against our form.io domain. This outage did not affect any of our Deployed customers.
- postmortem Aug 30, 2023, 02:44 PM UTC
The cause for this outage was triggered by a configuration change on our Database IP whitelist where an "internal" VPC IP address was removed from the database access white listing. This "internal" VPC IP address was mistaken as a temporary "debugging" whitelist address, but was in fact, the internal IP for the database within the VPC peering from our AWS deployment. This was the address that our instances use to connect to the database. As soon as this was removed, we started seeing failures in network activity and eventually our API's became unresponsive. We identified this error and made the corrections but the change took approximately 10 minutes to propagate which is why our network was down for that amount of time. Due to this incident, we will be making several changes and improvements to our process and deployments to ensure this kind of thing does not happen again. The first improvement we will make is a process improvement, where any changes to any configurations to our database \(regardless of how benign it seems\) will first be done within our test deployment and database to ensure it works as expected and does not have any side effects. The second improvement we will make will be to slowly migrate some of our intensive load bearing customers onto their own separate deployment so that their traffic and intensive use does not affect other hosted enterprise customers.