Production - AutoSubmits not processing as expected

Update timeline

1. identified Oct 22, 2021, 03:44 PM UTC

   We are investigating an issue whereby autosubmits are not processing as expected for customers. SimpliGov has identified the issue and is working on remediation.

2. monitoring Oct 22, 2021, 04:22 PM UTC

   A fix has been implemented and we are continuing to monitor the situation.

3. resolved Oct 22, 2021, 05:32 PM UTC

   This incident has been resolved and autosubmit functionality is working as expected. We will be posting a post-mortem for this incident on our status page shortly.

4. postmortem Oct 22, 2021, 07:01 PM UTC

   **Preliminary Root Cause:** SimpliGov auto-submits rely upon backend submission services hosted on Azure Service Fabric and utilizing Azure Service Bus for processing, queuing etc. The SimpliGov developed submit processing services used to process these autosubmits stalled on previously incomplete auto-submits which were queued for batch processing between 6 AM PST and 7 AM PST \(batching occurred at different times for different customers\). The batch sizes today were larger as a result of processing backlogs from the SimpliGov incident noted on 10/21/2021. During this period of time, intake submissions should have worked as expected for all async processes and most synchronous workflow configurations. In cases where chained auto-submits with zero delay would have provided users with a different user experience than they would be accustomed to. In such cases, users would have been shown a thank you page from the initial relationship rather than the last relationship in the autosubmit chain SimpliGov identified the autosubmit records that caused this issue, removed them from the queue and restarted all affected customer processing services. From 9:15 AM PST, auto-submits began processing as expected and customer backlogs and batches were cleared within ~15 minutes depending on the number of auto-submits used within a customers tenants. The more auto-submits used by a customer, the longer it took for said auto-submit backlog to be processed. Whilst the backlog of auto-submits etc. was processing, customers may have experienced some lags on dashboard updates etc. whilst dashboard update services scaled to work with the immediate increase in dashboard update requests. **Mitigation:** In the immediate term, SimpliGov immediately restarted all backend processing services to remove any potentially hung SQL sessions and identified the problematic hung submissions in the database. After removing such records and restarting the submit servicesm they self-healed and began to process auto-submits as expected. Going forward, SimpliGov has implemented several new features within our upcoming Dec 2021/Jan 2022 Production release allowing for better failover and fault tolerance in such scenarios. SimpliGov will be hosted on Azure Kubernetes Services instead of Azure Service Fabric, split service bus queue architecture will be enabled and additional retry policies have been added to handle scenarios whereby Azure Services return transient errors. These 3 particular items, in addition to smaller individual fixes and improvements should reduce the propencity of such incidents going forward. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward with auto-submits and will be deploying additional updates / architectural changes as part of our upcoming production release scheduled for Jan 2022. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team.