SimpliGov incident
502 Gateway Unavailable errors reported - Preproduction
SimpliGov experienced a major incident on February 15, 2022 affecting SimpliGov PreProduction, lasting 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 15, 2022, 12:26 AM UTC
SimpliGov support are investigating 502 status code errors reported on preproduction.
- resolved Feb 15, 2022, 12:38 AM UTC
The incident has been resolved and the preproduction environment is now available to users
- postmortem Feb 15, 2022, 12:40 AM UTC
**Preliminary Root Cause:** SimpliGov utilizes Azure App Configuration and key vault for secret sharing with authentication services. The key required for authentication between SimpliGov services rotated and stalled during rotation, causing the 502 status code errors noted. **Mitigation:** SimpliGov support manually rotated the key required and have gathered logs to investigate the matter further **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward and will identify the underlying rotation stall issue. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.