GoFormz experienced a major incident on December 3, 2020 affecting Web App and Mobile Sync API, lasting 3h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 03, 2020, 03:12 PM UTC
GoFormz Engineering is investigating a partial outage of the GoFormz Web App affecting areas including some Template editing actions, access to the Workflow page, and Users & Groups actions.
- investigating Dec 03, 2020, 03:32 PM UTC
We are continuing to investigate this issue.
- monitoring Dec 03, 2020, 04:16 PM UTC
GoFormz Engineering has identified the cause of this issue and implemented a fix. We are continuing to monitor all services.
- resolved Dec 03, 2020, 06:50 PM UTC
Starting at approximately 9 pm PST on Wednesday, December 2nd, a corrupt Azure instance within the GoFormz Gateway service began causing isolated degraded performance in the GoFormz Web App and Mobile Sync performance. After standard recovery operations did not immediately resolve the issue, GoFormz Engineering implemented startup configuration changes that returned the service to normal operations. As mitigation steps, GoFormz has verified service startup configurations, are adding additional monitoring, and investigating Azure instance instability with Microsoft Azure support.