Production - Autosubmits not processing/processing with delay

Update timeline

1. identified Dec 13, 2021, 11:29 AM UTC

   We are noting higher than expected levels of auto-submits not processing within our production environments. After raising a ticket with Azure Support, it has been noted that Azure experienced an internal issue within it's primary government environment which resulted in failed certificate rotation and hence, degradations in services used by SimpliGov, specifically DNS and Service Fabric. SimpliGov support is actively working on resolving the issue with the Azure support team and will process any backlogs of auto-submits as soon as possible. A full root-cause analysis will be provided by Azure support and this information will be provided to customers when it comes available.

2. monitoring Dec 13, 2021, 02:00 PM UTC

   Azure support have resolved the issue in conjunction with SimpliGov support and auto-submits are processing as expected. SimpliGov is continuing to monitor the situation whilst also running dashboard syncing processes to ensure that all dashboards reflect the true and correct statuses for customer records. A full RCA will be made available as soon as further information is received from Azure support regarding the failed certificate rotation task on their end.

3. resolved Dec 13, 2021, 02:48 PM UTC

   The incident has been resolved and dashboard syncs have completed for the majority of customer tenants. A full RCA will be published as soon as further information has been provided by Azure Support

4. postmortem Dec 17, 2021, 06:28 AM UTC

   **Preliminary Root Cause:** SimpliGov auto-submits rely upon backend submission services hosted on Azure Service Fabric and utilizing Azure Service Bus for processing, queuing etc. The SimpliGov developed submit processing services used to process these autosubmits experienced intermittent connectivity issues from 12/10/2021 to 12/12/2021. The batch sizes today were larger as a result of processing backlogs from the SimpliGov incident noted on 10/21/2021. During this period of time, intake submissions should have worked as expected for all async processes and most synchronous workflow configurations. In cases where chained auto-submits with zero delay would have provided users with a different user experience than they would be accustomed to. In such cases, users would have been shown a thank you page from the initial relationship rather than the last relationship in the autosubmit chain The root cause of the connectivity issue referenced was due to failed Azure maintenance procedures whereby DNS certificate in a US Government region failed automatic rotation before expiring. When the service was manually triggered to rotate, that operation failed as well. Azure engineering had to take down a DNS cluster that managed resolution for about 12 services in one capacity or another and manually swap certificates and redeploy the DNS cluster. Azure support are still determining the root cause of the rotation failure on their side and teams are actively working on it. Actions have been taken to increase the availability of DNS across Azure Gov at this time, though the details of that are not public and likely won't be due to security requirements around Azure topology. Whilst the backlog of auto-submits etc. was processing, customers may have experienced some lags on dashboard updates etc. as dashboard update services scaled to work with the immediate increase in dashboard update requests. **Mitigation:** SimpliGov worked with Azure support to determine the cause of the issue and resolve it during several support sessions. Additional capacity was added to expedite auto-submit processing for the backlog and dashboard sync processes were run to ensure that dashboards contained current and accurate data. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward with auto-submits and will be deploying additional updates / architectural changes as part of our upcoming production release scheduled for Jan 2022. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.