MetaQuery Dashboard Synchronization Delays - Production - 05/18/2021

Update timeline

1. investigating May 18, 2021, 09:18 PM UTC

   We are currently investigating slower than expected dashboard update times on a number of customer tenants. Users may note that submissions are not showing as expected in their dashboard whilst this is being resolved.

2. investigating May 18, 2021, 09:29 PM UTC

   We are continuing to investigate this issue.

3. identified May 18, 2021, 09:35 PM UTC

   The issue has been identified and we are syncing backlogged records to affected customer databases.

4. monitoring May 18, 2021, 09:44 PM UTC

   The dashboard sync has been completed for affected tenants and we are continuing to monitor the situation.

5. resolved May 18, 2021, 10:30 PM UTC

   All records have been synced to affected customer dashboards and the issue is resolved. A full post-mortem will be provided shortly.

6. postmortem May 19, 2021, 01:50 AM UTC

   **Preliminary Root Cause:** SimpliGov’s dashboard utilizes our proprietary MetaQuery ETL process and database as it’s underlying data source. In this case, a single record manually being synced caused a backlog in synchronization of other records with users experiencing a delay in information coming through on their dashboard. The backlog of updates to process on the dashboard caused customers to experience a delay in records being updated with appropriate statuses etc. While the backlog of dashboard updates was processing, customers continued to experience lags on dashboard updates. **Mitigation:** In the immediate term, SimpliGov immediately synchronized the backlog of records to be synced to the dashboard. Note that this only affected a single customer and if you were not contacted about dashboard delays, your tenant was not impacted by this incident. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor dashboard synchronization performance for the affected customer. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team.