SimpliSign Interactive Performance Degradation

Update timeline

1. investigating Jun 16, 2022, 08:43 PM UTC

   We are currently investigating a reported incident involving intermittent 502 Bad Gateway errors when accessing SimpliSign Interactive.

2. identified Jun 17, 2022, 05:22 AM UTC

   SimpliGov OPS have identified an issue within the Azure Application Gateway load balancer that routes requests to various SimpliSign services. The application gateway (load balancer) is recognizing that the applications behind it were up and running but the Azure Application Gateway still throws 502 errors and refuses to consistently route URLs correctly. SimpliGov OPS are working with Azure Support on this. We will provide an additional update when further information becomes available.

3. resolved Jun 17, 2022, 01:37 PM UTC

   Microsoft Azure support has resolved the issue on their end and provided a prelimiary root cause analysis for the cause of this issue. SimpliGov OPS have been monitoring the performance of SimpliSign since they pushed their update and the fix appears to have resolved all known issues regarding 502 status codes being shown to users. Additional information will be posted in the post-mortem for this incident.

4. postmortem Jun 17, 2022, 01:38 PM UTC

   **Summary:** _Cause:_ Azure introduced IPv6 addresses to the Azure Application Gateway that load balances traffic for SimpliSign. IPv6 addresses are not supported yet by Azure Application Gateway so redirection to these addresses resulted in a 502 status code for users _Manifestation:_ Users intermittently received 502 status code and error pages which prevented them from accessing SimpliSign signing pages, SimpliSign interactive pages etc. _Resolution:_ Microsoft Azure Support restarted backend services on their side and confirmed that DNS resolution does not occur for invalid IPv6 configurations. **Preliminary Root Cause:** SimpliGov utilizes Azure Application Gateway for load balancing SimpliSign services for all customers. This infrastructure component decides which application is used for each request and routes requests to different backend servers/applications based on the overall load within the system. It was noted that despite having healthy, available backend applications, the Azure Applicatoin Gateway was still throwing 502 status code errors even though all health checks and communication tests between the Azure Application Gateway and the backend SimpliSign applications were passing as expected. The outcome of the previous was that SimpliSign pages were intermittently unavailable. SimpliGov enlisted Azure Support and their Product Group to assist with resolving this issue. After debugging sessions with both Azure Support and the Azure Product Group, it was determined that a recent update by the Microsoft Azure Government team caused the issue. Azure Support released an update to apply IPv6 configurations to Azure Application Gateways within the Microsoft Azure Government cloud. During their update, the Azure Product Group restarted the components that make up the Azure Application Gateway but in one case, their restart processes did not have the desired effect in making sure that correct DNS resolution was applied for the backend applications behind the Application Gateway. This led to some components of the Application Gateway having multiple IPv6 and IPv4 backends. Azure Support confirmed that the presence of IPv6 backends was the direct cause of this issue as IPv6 is not supported for Azure Application Gateway backends. The presence of both IPv6 and IPv4 addresses explains the intermittent nature of this issue in that redirection to IPv4 backends resulted in successful interactions while interactions with IPv6 backends resulted in a 502 status code and error page being shown to the user. **Mitigation:** Azure Support restarted the components that held IPv6 IP addresses to make sure that they were removed and correct DNS resolution could take place. In simpler terms, Azure Support restarted the Application Gateway and made sure that only IPv4 addresses were present as IPv6 is not supported by the Azure Application Gateway currently. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward and will identify the underlying rotation stall issue. SimpliGov will continue working with Azure Support to try and prevent such occurences in the future. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.