Production - Degraded submission and dashboard performance - 2021/10/21

Update timeline

1. monitoring Oct 21, 2021, 07:55 PM UTC

   SimpliGov has been notified of degraded dashboard and submission performance by a subset of customers. SimpliGov has identified the issue and reached out to our hosting provider, Azure Government. Azure Government has confirmed that they are working on said issue and will report back periodically regarding the status of their investigation, resolution etc. During this period of time, please note that submissions are still being processed but there may be a lag between submission creation / updates and what is shown on the dashboard. SimpliGov support personnel have activated bypass procedures to ensure that records are being shown on customer dashboards while Azure Government resolve the service issue on their end.

2. monitoring Oct 21, 2021, 08:40 PM UTC

   Azure support have confirmed that they have resolved the issue and are preparing a root cause analysis report. SimpliGov will share further information as it is made available. Processing and Dashboard Update services are working as expected but SimpliGov support are continuing to reconcile older records on the dashboard.

3. monitoring Oct 21, 2021, 08:47 PM UTC

   Azure support has provided a preliminary root cause analysis regarding the increased error rate SimpliGov received for Service Bus connections: Summary of impact: Between 12:00 and 13:00 EDT on 21 OCT 2021, Azure Government identified SimpliGov as a customer using Service Bus in USGov Arizona who may have experienced increased error rates from your service bus instance Preliminary root cause: Azure Government determined that a subset of backend instances experienced an unexpected high utilization due to a platform upgrade in US Gov Arizona. Mitigation: The Azure Government product engineering group allocated more bandwidth for the host nodes which brough the instances back to a healthy state Next steps: Azure Government will continue to investigate to establish the full root cause and prevent future occurrences. A full root cause analysis is expected from Azure Government in approximately 2 to 3 business days.

4. resolved Oct 21, 2021, 11:04 PM UTC

   This incident has been resolved. A post-mortem with follow up actions will be posted shortly.

5. postmortem Oct 22, 2021, 07:04 PM UTC

   **Preliminary Root Cause:** SimpliGov utilizes Azure Service Bus as it’s message queuing system for things like submissions, dashboard updates etc. Azure Government support informed SimpliGov that they performed updates to Azure Service Bus infrastructure and between 9:00 and 10:00 PST on 21-10-2021, SimpliGov was identified as a customer using Service Bus in USGov Arizona which experienced increased error rates for service bus. SimpliGov received “Connection reset by peer” error messages from the Azure Service bus service which resulted in partial connection loss to the service. Azure support states that the primary cause of the issue on their side was that a subset of backend instances experienced an unexpected high utilization due to a platform upgrade in US Gov Arizona. The Azure Government product engineering group allocated more bandwidth for the host nodes which brough the instances back to a healthy state. From the SimpliGov side, this mitigating action allowed our services to connect to Azure Service bus as expected. During this period of time, most intake submissions should have worked as expected but some users would have experienced slower than expected dashboard synchronization times and potential duplicates depending on their workflow configuration as submissions were retried when initial connections to service bus failed. **Mitigation:** In the immediate term, SimpliGov support employed manual dashboard synchronization processes to ensure that any records coming in were being reflected as soon as possible. Azure support resolved the Service Bus issue on their end and this allowed SimpliGov processing and dashboard synchronization services to function as expected. Going forward, SimpliGov has implemented several new features within our upcoming Dec 2021/Jan 2022 Production release allowing for better failover and fault tolerance in such scenarios. SimpliGov will be hosted on Azure Kubernetes Services instead of Azure Service Fabric, split service bus queue architecture will be enabled and additional retry policies have been added to handle scenarios whereby Azure Services return transient errors. These 3 particular items, in addition to smaller individual fixes and improvements should reduce the propencity of such incidents going forward. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward with azure service bus and configure specific alerts relating to Service bus and “connection reset by peer' messages. We will be deploying additional updates / architectural changes as part of our upcoming production release scheduled for Jan 2022 to also improve future fault tolerance. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team. ------------------------------------------------------------------------------------------------------------------------------------------------------- _Additional RCA from Microsoft Azure Government support:_ **Incident Summary:** Between 10/21 17:07 UTC and 10/21 19:14 UTC, Event Hub, Service Bus, and Relay service customers may have intermittently experienced an increase in latency or timeouts on runtime operations. **Root Cause:** On 10/21/21 at 17:07 UTC, an upgrade was performed on clusters servicing Service Bus, Event Hub, and Relay services. During the upgrade, as TCP connections were disconnected from each gateway machine, there was a noticeable increase in TCP connection requests as existing TCP connections were terminated resulting in high CPU utilization by the LSASS process that handles TLS/SSL handshakes. As the LSASS process hit the CPU threshold set by the service teams, processing of TLS/SSL handshakes were slowed causing increased latency/delay in serving send and receive requests made by client applications. The issue was mitigated at 10/21/21 at 19:14 UTC after the team scaled out the cluster and added machines to allocate more resources to process incoming requests. **Next Steps:** Microsoft apologizes for the impact to affected customers. We have scaled out our gateway machines to increase capacity and are taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes \(but is not limited to\): * Enhancements to monitoring related to LSASS CPU utilization * Performance improvements to the throttle/delay response related to invalid tokens/SAS * Adjustment of diagnostic steps to include tracing namespace/audience when clients prevent invalid tokens/SAS