SimpliGov incident
CDN not returning frontend resources for some users
SimpliGov experienced a major incident on July 30, 2024 affecting SimpliGov Production and SimpliGov PreProduction, lasting 4h 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 30, 2024, 01:10 PM UTC
SimpliGov has identified slow response times for CDN resources to certain customer locations. SimpliGov support are working with Azure infrastructure engineers to resolve this issue as soon as possible.
- identified Jul 30, 2024, 01:18 PM UTC
As of 06:13 AM PST, Azure Support have reported the following on their status page: Network Infrastructure - Issues accessing a subset of Microsoft services We are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services. We have multiple engineering teams engaged to diagnose and resolve the issue. More details will be provided as soon as possible. SimpliGov is continuing to work with Azure infrastructure engineers to assist with resolving this issue as soon as possible.
- identified Jul 30, 2024, 01:42 PM UTC
SimpliGov support are continuing to work with Azure support and monitor the site for improvements in CDN response times.
- monitoring Jul 30, 2024, 02:01 PM UTC
SimpliGov are still working with Azure support on this issue but have noted an improvement in CDN loading times etc. SimpliGov is continuing to monitor the situation.
- monitoring Jul 30, 2024, 02:33 PM UTC
SimpliGov are continuing work with Azure support on this issue but continue to note stable CDN resolution times etc. SimpliGov is continuing to monitor the situation.
- monitoring Jul 30, 2024, 03:03 PM UTC
We are continuing to monitor the situation but note that some customers are still affected. We will post another update as soon as new information becomes available
- monitoring Jul 30, 2024, 03:49 PM UTC
SimpliGov has begun migrating it's CDN endpoints to secondary infrastructure given the ongoing issues experienced on it's primary CDN infrastructure. The SimpliGov IT team have migrated both staging and preproduction environments and is in process with the production and training environments. Another update will be posted shortly when further information becomes available
- monitoring Jul 30, 2024, 04:14 PM UTC
SimpliGov's preproduction, training and staging environments have migrated to secondary infrastructure and customers should see a return to normal behavior across all sites. Production migration is ongoing. An additional update will be posted when further information is available.
- monitoring Jul 30, 2024, 04:51 PM UTC
SimpliGov's Production, preproduction, training and staging environments have migrated to secondary infrastructure and customers should see a return to normal behavior across all sites. The SimpliGov team is continuing to monitor performance on the CDN and will post additional updates shortly.
- resolved Jul 30, 2024, 05:59 PM UTC
This incident has been resolved and production, preproduction, training and staging services have been fully restored.