SimpliGov experienced a minor incident on January 18, 2022, lasting 1h 22m. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Jan 18, 2022, 06:16 PM UTC
Some customers within our production environment may be experiencing longer than expected submission times. We have identified the cause of the issue.
- identified Jan 18, 2022, 06:16 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Jan 18, 2022, 07:08 PM UTC
A fix has been implemented and the backlog has been processed. SimpliGov OPS are continuing to monitor the situation closely.
- resolved Jan 18, 2022, 07:39 PM UTC
This incident has been resolved. A root cause analysis will be published to the SimpliGov status page later today.
- postmortem Jan 19, 2022, 02:23 PM UTC
**Preliminary Root Cause:** SimpliGov auto-submit retry batching occurred during a recent period of instability, causing a significant backlog of records to process across all customers. Due to the backlog size and time to horizontally scale, customers experienced degraded submission times and potential 504 errors on submission. **Mitigation:** SimpliGov support determined the cause of the issue and manually scaled capacity by adding more nodes and additional submission pods to expedite auto-submit processing for the submission backlog. Dashboard sync processes were run to ensure that dashboards contained current and accurate data. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.