GoFormz incident
Partial Disruption of GoFormz Services for Some Accounts
GoFormz experienced a major incident on December 19, 2021 affecting Web App, lasting 4h 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 19, 2021, 04:25 PM UTC
GoFormz Engineering is currently investigating a partial disruption to GoFormz Services for some accounts. Actions may be unavailable or exhibit degraded performance.
- investigating Dec 19, 2021, 04:43 PM UTC
We are continuing to investigate this issue.
- monitoring Dec 19, 2021, 04:48 PM UTC
GoFormz Engineering has identified the cause of this issue and all web app services are now operational. We are continuing to monitor all services.
- resolved Dec 19, 2021, 08:51 PM UTC
This incident has been resolved.
- postmortem Dec 21, 2021, 06:06 PM UTC
**Timeline**: Sunday 12/19/21 6:28 am - 8:32am PST **Root Cause Analysis**: After a detailed investigation, the GoFormz Engineering team determined that one of the database pools had insufficient storage space, due to unreleased data during our normal database maintenance. This resulted in a lack of space available for a subset of customers. **Mitigation**: Immediate action was taken to increase the available space on the database pool. Additionally, alerts were updated to include total available storage space.