MetaQuery Dashboard and "Queryable Workflow Instance" API Endpoint Synchronization Delays - Production - 06/04/2021

Update timeline

1. resolved Jun 08, 2021, 11:10 PM UTC

   Between 2:30 AM PST and 1:58 PM PST on 06/04, Azure reported issues with connections to existing databases in the secondary region experiencing errors or timeouts, and existing connections may have been terminated. Issues replicating from our primary to secondary databases caused slower than expected transfer of data to our MetaQuery databases. This impacted the timeliness with which records were synced to the SimpliGov dashboard and the "Queryable Workflow Instance" API Endpoint

2. postmortem Jun 08, 2021, 11:11 PM UTC

   **Preliminary Root Cause:** SimpliGov’s dashboard and "Queryable Workflow Instance" API Endpoints utilize SimpliGov's MetaQuery ETL process and database as it’s underlying data source. All MetaQuery databases are geo-replicated to a secondary region, US Gov Texas in Azure Government using Azure's built in geo-replication functionality for Azure SQL. Between 2:30 AM PST and 1:58 PM PST on 06/04, Azure reported issues with connections to existing databases in the secondary region experiencing errors or timeouts, and existing connections may have been terminated. Issues replicating from our primary to secondary databases caused slower than expected transfer of data to our MetaQuery databases. Azure Government noted that a recent change on their DNS registry contained a configuration error in which the cluster hosting the nodes became unhealthy, leading to control plane failures and connection issues. The issue was mitigated by the Azure Government team by rolling back the configuration change noted. The backlog of updates to process on the dashboard caused customers to experience a delay in records being updated with appropriate statuses etc. While the backlog of dashboard updates was processing, customers continued to experience lags on dashboard updates. **Mitigation:** In the immediate term, SimpliGov manually synchronized any records with long synchronization times and monitored sync processes during the time period noted. After receiving notice from the Azure Government team that the issue had been mitigated, any backlog of records to be synced to the MetaQuery database was processed and APIs should have been returning appropriate results. If you were not directly contacted about dashboard delays or invalid responses to the "Queryable Workflow Instance" API endpoint, your tenant was not impacted by this incident. Next Steps: We apologize for the impact to affected customers. SimpliGov will continue to monitor dashboard synchronization performance for the affected customer. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team. Should customers have a critical reliance or dependency on the SimpliGov "Queryable Workflow Instance" API Endpoint for retrieving workflow tokens etc., they may want to consider using the @WorkflowTokenId formula to pull the latest TokenID for an open workflow record. This method will also remove any potential delays resulting from the asynchronous transfer of data from our transactional databases to customer MetaQuery databases.