Degraded Submission Service Performance - Production - 05/17/2021

Update timeline

1. investigating May 18, 2021, 03:08 AM UTC

   We are currently investigating degraded submission service performance on a number of customer tenants. Users may note that submissions are not showing as expected in their dashboard whilst this is being resolved.

2. identified May 18, 2021, 03:20 AM UTC

   The issue has been identified and resolved. We are continuing to monitor the situation but all services are back online and processing as expected.

3. monitoring May 18, 2021, 03:45 AM UTC

   The issue has been resolved and we are continuing to monitor the situation.

4. resolved May 18, 2021, 04:39 AM UTC

   The incident has been resolved and a full post-mortem will be provided soon. We will contact customers separately if any follow up actions are required.

5. postmortem May 18, 2021, 04:40 AM UTC

   **Preliminary Root Cause:** SimpliGov utilizes Azure Government service fabric for hosting submission services within the platform. Service fabric utilizes virtual machine scale sets with nodes being added and removed from the virtual machine scale set when auto-scaling operations are run. During a recent auto-scaling process, the nodes deprovisioned from the virtual machine scale set caused the transition of submission services between nodes with some not initializing correctly. The SimpliGov developed submission processing services did not respond in a timely manner because of this incorrect initialization and this caused customers to experience 504 status code responses, indicating that submission processing was not occurring as quickly as expected. SimpliGov restarted the affected submission services and the backlog of submissions built up during the incident period were processed as expected. Whilst the backlog of submits, auto-submits etc. was processing, customers may have experienced some lags on dashboard updates etc. whilst dashboard update services scaled to work with the immediate increase in dashboard update requests. **Mitigation:** In the immediate term, SimpliGov immediately restarted all backend processing services to initialize the submission services affected. The auto-scaling rules configured have been reconfigured to have a higher threshold for scaling down operations also. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the submission services. All customer records being processed throughout the downtime event should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team.