Benevity experienced a major incident on July 17, 2025 affecting Grants Advanced (formerly Versaic), lasting 15s. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 17, 2025, 03:18 PM UTC
We are currently investigating the issue. Some applications such as Budget Module, Proposal Collaboration, Mass actions are impacted.
- resolved Jul 17, 2025, 03:19 PM UTC
This incident has been resolved.
- postmortem Aug 01, 2025, 08:25 PM UTC
### Summary At 2:00 a.m. UTC on July 17, 2025, a partial service impairment occurred in the Versaic platform, impacting Payment Management, Background Tasks, and Proposal Collaboration features for clients using the third-party Budget Module. The issue was identified through internal monitoring, and mitigation efforts began immediately. A resolution was implemented, and full system functionality was restored by 9:00 a.m. UTC on the same day. ### Impact Between 2:00 a.m. and 9:00 a.m. UTC on July 17, 2025, clients using the third-party Budget Module in Versaic may have encountered errors and inoperable functionality when accessing Payment Management or Proposal Collaboration workflows. As part of the recovery process, the engineering team reprocessed all background tasks and jobs affected during the outage to ensure full and accurate system restoration. ### Root Cause The incident was caused by a configuration update that unintentionally disrupted the Versaic system’s access to its database. The changes, deployed independently of the application code, were mistakenly applied across all environments due to a bug in the configuration script. ### Future Mitigation 1. The Versaic configuration method will be updated to align with code deployment. 2. The CI/CD pipeline will be updated to ensure a connectivity test after each configuration update for role-based operability. 3. Maintain, track and deploy all configuration settings in code \(via Terraform\) for greatest environment isolation. ### Timeline of Events * July 16, 2025 \(4 PM UTC\): Configuration change was created. * July 17, 2025 \(2 AM UTC\): The system picked up the new configuration, issues begin. * July 17, 2025 \(9 AM UTC\): The configuration was reverted.