SimpliGov experienced a major incident on April 20, 2021, lasting 7h. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Apr 21, 2021, 10:30 AM UTC
Summary of Impact: Between 2:05 PM PST on April 20th 2021 to 2:27 PM PST on April 20th 2021, all customers using the preproduction environment experienced issues accessing SimpliGov pages, making API calls to SimpliGov and using the service in general. Users were presented with a 502 status page indicating that no backend servers were available to service frontend requests.
- postmortem Apr 21, 2021, 10:30 AM UTC
**Preliminary Root Cause:** As a result of an upgrade activity initiated by the Azure Government team, Service Fabric clusters used to provide the preproduction services failed to retrieve node statuses from the underlying virtual machine scale sets. The upgrade pushed by the Azure Government team was expected to apply the latest fabric updates to the cluster on each node sequentially, applying the update on a single seed node per fault domain within the Azure Service Fabric cluster. After applying the upgrade, the seed nodes required for the service fabric clusters failed to report their status to the Service Fabric management services and the Service Fabric cluster could not service any requests for SimpliGov’s Portal. Authentication, API, MetaQuery Sync and Submission services. SimpliGov customers identified the downtime event by trying to access the SimpliGov portal via web browser and received a 502 status code page indicating that the Azure Application Gateway used for load balancing requests to SimpliGov Service Fabric clusters was not receiving appropriate health status responses to allow the Application Gateway to service requests to customers. **Mitigation:** In the immediate term, SimpliGov restarted all nodes in all virtual machine scale sets powering the Service Fabric clusters at approximately 2:15 PM PST. This triggered roll back procedures for the Service Fabric upgrades initiated by the Azure Government team and the roll back completed at approximately 2:27 PM PST at which point, SimpliGov services became available for customers. In addition to working to restore services as soon as possible, SimpliGov contacted the Azure Government support team to request assistance with the roll back if required and a full root cause analysis on why the upgrades failed to apply sequentially and through each fault domain as expected. Information was also requested on how the upgrade caused all nodes in the underlying Virtual Machine Scale Sets to fail to report appropriate statuses to the Service Fabric management services. Azure Government support also confirmed the successful roll back of the upgrade to the last known successful version of the Service Fabric services. Outside of US business hours, SimpliGov IT/Operations personnel manually applied the latest Service Fabric upgrades and monitored their successful application throughout and after the upgrade process. It was noted that the upgrades were applied to nodes in a sequential manner as expected with no effect to the services used by customers. **Next Steps:** We apologize for the impact to affected customers. Azure Government will provide additional details on why the upgrade process didn’t apply in the expected manner, caused the nodes to fail to report to the Service Fabric management service and why a roll back of the upgrade was required to bring SimpliGov services back online. These additional details will be in the form of a Root Cause Analysis report. Note that as this event occurred on production with submission, API and portal services being unavailable, all customer records being processed throughout the downtime event should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their preproduction tenants.