SimpliGov - Production 502 Status Code observed

Update timeline

1. investigating Oct 05, 2023, 03:14 PM UTC

   502 errors are being observed by some users at this time. SimpliGov IT have identified the issue and are resolving it.

2. identified Oct 05, 2023, 05:10 PM UTC

   We have identified the cause of the performance issue. Customer may still experience performance issues and potential 502s while we resolve said problem.

3. monitoring Oct 05, 2023, 08:19 PM UTC

   SimpliGov has resolved the issue in conjunction with Azure Engineers. We are continuing to monitor behavior before closing this incident out as resolved. A full root cause analysis will be published on our status page.

4. resolved Oct 05, 2023, 11:00 PM UTC

   Firstly, we want to apologize for the performance issues noted and any inconvenience caused by them. SimpliGov takes these matters very seriously and has pulled in all appropriate team members to ensure that they have been addressed. Upon conducting a thorough investigation, we identified that the root cause of the issue was associated with Azure CDN. Our process of elimintation involved ruling out backend and frontend changes made as part of the September 2023 release and after performing trace analysis for load times on critical JavaScript files and other static assets, it was determined that the CDN was the cause of the issue. Azure CDN is important to the SimpliGov platform as it hosts JavaScript, Image, CSS and font files which are used throughout all areas of the SimpliGov portal. SimpliGov support have direct subscriptions to Azure service disruption notifications but did not receive any such notifications from the Azure support team. However, after contacting support to flag this high severity case, Azure support have acknowledged that they have been experiencing support issues and errors within their CDN services as a result of an upcoming change from Akamai CDN to Microsoft CDN. Once Azure support addressed the CDN issues, performance improved within the SimpliGov platform and errors noted in our telemetry dropped dramatically. SimpliGov support continues to monitor system behavior but we believe that the platform is, and continues, to operate as expected. In addition to working with the Azure support team, SimpliGov is working with it's dedicated Azure CXP (Customer Experience) team and Azure Product team for further analysis of why their preparations caused issues with existing CDN endpoints. Once this information becomes available, it will be added to our Status Page postmortem at SimpliGov LLC Status To prevent such occurrences in the future, SimpliGov support will now manually migrate from Azure Akamai CDN to Azure Microsoft CDN in the near term and remove the requirement for Azure support to perform the migration on their side. This migration activity will be scheduled in due course and may require a small amount of downtime while assets are moving from the Akamai CDN to the Microsoft CDN. The following link serves notice that Akamai CDN is being deprovisioned with a cutover to Microsoft CDN on October 31st 2023 (https://learn.microsoft.com/en-us/azure/cdn/cdn-features). Once again, we deeply apologize for any inconvenience this may have caused you. Your satisfaction is of utmost importance to us, and we are committed to providing you with a seamless experience moving forward. If you have any further questions or concerns, please do not hesitate to reach out to us via Service Desk or via your dedicated Professional Services and Account Management team member.