GoFormz experienced a minor incident on December 3, 2021 affecting Web App and API (v2), lasting 6d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Dec 03, 2021, 04:37 PM UTC
We are currently monitoring degraded performance in our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.
- monitoring Dec 03, 2021, 10:32 PM UTC
GoFormz Engineering implemented improvements to the PDF export service and all operations have returned to normal. We are continuing to monitor this incident.
- investigating Dec 06, 2021, 04:47 PM UTC
GoFormz Engineering is investigating the degraded performance of our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.
- monitoring Dec 06, 2021, 06:14 PM UTC
We are currently monitoring degraded performance in our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.
- resolved Dec 09, 2021, 06:22 PM UTC
This incident has been resolved.
- postmortem Dec 09, 2021, 08:51 PM UTC
**Summary**: On 11/16, code containing a defect was released. This code was not in use until 12/02, when customers attempting to generate a pdf saw service errors or latency. **Root Cause Analysis**: After a detailed investigation, the GoFormz Engineering team determined that there was an underlying code defect in a 3rd-party pdf tool that caused a higher-than-normal amount of CPU. During high traffic, this defect created a deadlock and resulted in several servers stalling when attempting to generate a pdf. Consequently, our API and Web services experienced errors and timeouts, and our backend queues experienced an increase in service latency. **Mitigation**: Immediate action was taken to scale out the pdf services, allowing for the increase in CPU to be handled. Complete mitigation was accomplished when the engineering team was able to correct the code defect and deploy a hotfix to all pdf services. **Next steps**: While this issue is resolved, GoFormz Engineering is continuing to closely monitor each service for errors or instability.