SimpliGov experienced a major incident on August 5, 2021 affecting SimpliGov PreProduction and SimpliSign PreProduction, lasting 2h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 05, 2021, 12:19 PM UTC
We are currently availability issues for our preproduction environment. We will post updates as they come available.
- identified Aug 05, 2021, 01:40 PM UTC
The issue has been identified and the IT/OPS team are actively working to resolve said issue. As a result of the fix, users will need to update their whitelists for non-production environments. Whitelists referencing 52.244.80.86 should be updated with a new IP address: 52.244.79.177.
- monitoring Aug 05, 2021, 02:54 PM UTC
The issue has been resolved and the preproduction environment is now working as expected. SimpliGov IT/OPS will continue to monitor the situation going forward.
- resolved Aug 05, 2021, 03:10 PM UTC
Preproduction services have been restored and are fully operational.
- postmortem Aug 05, 2021, 03:59 PM UTC
**Executive Summary:** Between 5:19 AM PST and 7:54 AM PST, SimpliGov preproduction was unavailable for customers causing users to receive pages stating "502" errors. As a result of this outage, users of affected customer websites were unable to submit new forms or work with existing forms in the preproduction environment. The cause of this partial outage was that periodic cluster updates failed to apply correctly and automated rollback processes on the cluster stalled. At 7:54 AM PST, normal service was resumed for the preproduction environment. After working with our hosting provider \(Azure Government\), it was determined that the best course of action was to redeploy the cluster and the services running on it. As a result of redeploying the cluster, the IP address for preproduction changed from 52.244.80.86 to 52.244.79.177. Customers should replace the old IP address \(52.244.80.86\) with the new IP address \(52.244.79.177\) if they are whitelisting communication from SimpliGov preproduction to internal systems. We apologize for the impact to affected customers. A more detailed summary of events can be seen below. **Detailed Summary:** Preliminary Root Cause: As a result of a long running upgrade activity on our preproduction Service Fabric cluster, Service Fabric initiated automatic rollback procedures to revert the updates applied and move back to the last successful version applied. The rollback activity noted ran for much longer than expected and was deemed "stalled". In the aforementioned scenario, the service fabric cluster could not service any requests for SimpliGov’s Portal. Authentication, API, MetaQuery Sync and Submission services and remained in a stalled status. SimpliGov users trying to access the preproduction via web browser received 502 status code pages indicating that the Azure Application Gateway used for load balancing requests to SimpliGov Service Fabric clusters was not receiving appropriate health status responses to allow the Application Gateway to service requests to customers. Likewise, API calls to preproduction received a 502 status code. Mitigation: After discussing available options with the Azure Government support team, SimpliGov redeployed the cluster affected and subsequentially redeployed all preproduction services affected. In addition to working to restore services as soon as possible, the Azure Government support team are working to identify why the upgrades and rollback processes failed. As a result of this mitigation strategy, customers should update any IP whitelists they have for non-production environments by replacing Preproduction's old IP address \(52.244.80.86\) with the new Preproduction IP address \(52.244.79.177\) Next Steps: We apologize for the impact to affected customers. Azure Government will provide additional details on why the upgrade and rollback processes didn’t work as expected, causing preproduction to become unavailable. Note that as this event occurred on preproduction with submission, API and portal services being unavailable, all customer records being processed throughout the downtime event should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their preproduction tenants.