GoFormz experienced a critical incident on April 17, 2020 affecting Web App and Automated Workflows and 1 more component, lasting 3h 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 17, 2020, 07:44 PM UTC
GoFormz is currently experiencing degraded performance. This primarily affects accounts using the GoFormz Workflow or API, and users not already logged into a GoFormz account through the mobile or web app.
- identified Apr 17, 2020, 08:26 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 17, 2020, 08:36 PM UTC
We have identified the issue and implemented a fix. As of approximately 1:20 PM PDT, GoFormz services have returned to normal functionality. We are continuing the closely monitor this issue.
- resolved Apr 17, 2020, 10:57 PM UTC
This incident has been resolved.
- postmortem Apr 17, 2020, 10:57 PM UTC
Essential components of the GoFormz platform became unresponsive on Friday 12:25pm Pacific Time due to a corrupt configuration change in our Authentication service. After identifying the underlying issue, GoFormz engineering rolled back the change and restarted impacted services. Upon service recovery GoFormz’s primary authentication database required upscaling to recover completely. All GoFormz services have returned to normal levels as of 1:20pm Pacific Time. GoFormz engineering will be reviewing and adjusting these configuration change procedures to mitigate future occurrences.