504 Errors Reported: Submission, Auto-Submit, Reassignment, Orchestrator activities failing on Production

Update timeline

1. investigating Apr 28, 2021, 05:15 PM UTC

   We are aware of various reports that customers are experiencing 504 gateway errors, slow load times, inability to process records, and inability to pull tasks for Orchestrator in the Production tenants. Our team is working on having this issue resolved as our top priority. If you or your users are experiencing any problems with SimpliGov on your Production environment please report it in a support desk ticket so we can gather all the information and resolve the issue as soon as possible. Submission services have degraded/crashed for some tenants because of certain looping submissions. The trigger event for this appears to be Azure Service Bus connectivity issues. We will keep you informed as things progress. Our sincerest apologies for any disruptions or inconvenience caused.

2. identified Apr 28, 2021, 08:00 PM UTC

   The issue has been identified and our team is working to resolve the issue.

3. monitoring Apr 28, 2021, 08:01 PM UTC

   A fix has been implemented and the SimpliGov team are testing and monitoring to confirm the platform is working as expected.

4. resolved Apr 28, 2021, 08:03 PM UTC

   The incident has been resolved since 11:55 am PST and customers should see their backlog of submissions processing within their tenant. SimpliGov will provide a root cause analysis for the issue at a later date.

5. postmortem Apr 29, 2021, 05:28 AM UTC

   **Preliminary Root Cause:** SimpliGov utilizes Azure Government service bus for message queuing activities within the application. Message queuing acts as a way for submissions, auto-submits, reassignments and a host of other activities to get their information into a queue for processing as soon as processing services are available. Upon attempting to retrieve messages from service bus message queues, SimpliGov received several bad responses and this created some backlogged items within submission tables in SimpliGov databases. The SimpliGov developed submit processing services used to process these messages from the queues crashed and this caused customers to experience 504 status code responses, indicating that submission processing was not happening in a timely manner. SimpliGov identified the submit records that were causing the submit processing services to crash as they were trying to enter duplicate submission records in the database, causing primary key constraint errors to be fired. SimpliGov fixed the statuses on the problematic records and the submit services self-healed at this point to allow the backlog of submissions during the affected period to process as expected. Whilst the backlog of submits, auto-submits etc. was processing, customers may have experienced some lags on dashboard updates etc. whilst dashboard update services scaled to work with the immediate increase in dashboard update requests. **Mitigation:** In the immediate term, SimpliGov immediately restarted all backend processing services to remove any potentially hung SQL sessions and identified the problematic hung submissions in the database. After updating the statuses of such records, the submit services which were previously crashing self-healed and began to process submits, auto-submits etc. As noted, SimpliGov have included a fix for this scenario in the upcoming May 8th 2021 release \(which has already been deployed to staging and preproduction environments as of March 28th 2021\). This update improves SimpliGov's submission service stability by avoiding the situation when an exception on a single tenant can affect other tenants processing. In order to prevent the situation of endless exception loop in the scenario noted, auto-submits with any exception will be immediately marked as “Failed” in the Auto-Submit dashboard from which tenant administrators can retry it. This will prevent any blocking submissions and individually failing auto-submits can be fixed as needed. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the fix applied and submission services and will be deploying an additional mitigating update as part of the May 8th 2021 production release. All customer records being processed throughout the downtime event should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team.