SimpliGov incident
Production slowness and calculation issues observed
SimpliGov experienced a minor incident on March 21, 2023 affecting SimpliGov Production and SimpliSign Production, lasting 1h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 21, 2023, 05:10 PM UTC
SimpliGov has observed unusual slowness and calculation issues for some customers. Our team is working on this and will report back with further information as soon as possible.
- identified Mar 21, 2023, 05:14 PM UTC
SimpliGov has identified the source of the issue and is working on resolving the issues noted.
- monitoring Mar 21, 2023, 05:26 PM UTC
SimpliGov has identified and resolved the underlying cause of the issue. SimpliGov OPS have updated the required settings to bring operations back to normal. SimpliGov OPS will continue to monitor the situation before providing a final root cause analysis and resolution status.
- resolved Mar 21, 2023, 06:50 PM UTC
SimpliGov OPS have monitored the behavior of the application for the last hour and can confirm that the application is responding as expected with usual calculation and dashboard performance. SimpliGov OPS are moving this to a resolved status with a post-mortem to follow for this incident.
- postmortem Mar 21, 2023, 06:51 PM UTC
**Preliminary Root Cause:** SimpliGov utilizes Azure SQL Databases for database shard management within the application. The “Shard Manager” used within Azure SQL experienced communication issues, resulting in timeouts. SimpliGov moved the database to a different server and updated settings relating to connections to the shard manager. **Mitigation:** SimpliGov support moved the database to another server and updated settings relating to connections to the shard manager. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward and will work with Azure support to understand and resolve the connection errors noted. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.