Degraded dashboard performance - Production - 2021/10/20

Update timeline

1. monitoring Oct 20, 2021, 07:32 PM UTC

   For a subset of customers, SimpliGov dashboards and API calls are showing submissions with a lag. This has been resolved and the underlying cause is being investigated further.

2. resolved Oct 20, 2021, 08:04 PM UTC

   This incident has been resolved. The noted downtime was around 2021-10-20 10:33 and 2021-10-20 10:37 i.e. this behavior was encountered for ~4 minutes. We have synced out any records affected during this time. Some customers may have continued to experience syncing issues beyond this period of time but intake and processing services for the dashboard were operational during this time. Regarding the cause of the outage, an underlying node was restarted by Azure updates (these are automated and not under SimpliGov’s control) and their failover manager service took some time to move some copies of customer services to other nodes. Should you have any further questions, please contact support via your Service Desk. Regards, SimpliGov OPS

3. postmortem Oct 22, 2021, 07:22 PM UTC

   **Preliminary Root Cause:** As a result of an upgrade activity initiated by the Azure Government team, Service Fabric clusters used to provide the production services failed to retrieve node statuses from the underlying virtual machine scale sets. The upgrade pushed by the Azure Government team was expected to apply the latest fabric updates to the cluster on each node sequentially, applying the update on a single seed node per fault domain within the Azure Service Fabric cluster. In cases of upgrade failure or node failure, it is expected that Azure Service Fabric will move services from affected nodes to active or "live" nodes. After applying the upgrade, a seed node required for the service fabric clusters failed to report their status to the Service Fabric management services and the Service Fabric cluster could not service any requests for SimpliGov’s Portal. Authentication, API, MetaQuery Sync and Submission services from this node. SimpliGov customers identified the downtime event by trying to access the SimpliGov portal via web browser and received a 502/504 status code page indicating that the Azure Application Gateway used for load balancing requests to SimpliGov Service Fabric clusters was not receiving appropriate health status responses to allow the Application Gateway to service requests to customers. **Mitigation:** Most services self-healed once the Service Fabric failover manager failed over services from the affected node to an active node. However, some dashboard synchronization services were affected post-healing event. In the immediate term, SimpliGov restarted the affected node containing dashboard synchronizatin services in the production virtual machine scale set, triggering move procedures for services originally hosted on the affected node. At 12:30 PM PST, the move activity completed and normal service was removed. In addition to working to restore services as soon as possible, SimpliGov contacted the Azure Government support team to request assistance with the issue and a full root cause analysis on why the upgrades failed to apply correctly and through each fault domain as expected. Information was also requested on how the upgrade caused all nodes in the underlying Virtual Machine Scale Sets to fail to report appropriate statuses to the Service Fabric management services. **Next Steps:** We apologize for the impact to affected customers. Azure Government will provide additional details on why the upgrade process didn’t apply in the expected manner, caused the nodes to fail to report to the Service Fabric management service and why the node failure did not trigger expected move processes without manual interaction from SimpliGov. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team.