Benevity experienced a major incident on March 3, 2025 affecting Grants Advanced (formerly Versaic), lasting 23h 54m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 03, 2025, 07:11 PM UTC
We have identified that we are are seeing a slower than normal response from the Versaic UI. We are investigating the cause and looking at remediation steps.
- identified Mar 03, 2025, 07:41 PM UTC
We have implemented a workaround, and are seeing improvement in response within the Versaic application. We are working on a permanent fix.
- identified Mar 03, 2025, 08:58 PM UTC
We are currently running our build pipeline to deploy a hotfix for this issue. We will post again when our deployment pipeline is complete and we have verified our fix.
- identified Mar 03, 2025, 11:17 PM UTC
We have deployed a fix, and we are currently monitoring and reviewing system performance.
- monitoring Mar 04, 2025, 01:40 AM UTC
A workaround has been deployed that appears to have alleviated the problem. We are continuing to monitor.
- resolved Mar 04, 2025, 07:05 PM UTC
This incident has been resolved.
- postmortem Apr 01, 2025, 12:10 AM UTC
## Summary On March 3, 2025 from 10:23 MT to 12:19 MT, users accessing Community Investment - Enterprise experienced slow page load times and occasional page timeouts as a result of application changes introduced that day. ## Impact Between 10:23 MT and 12:19 MT any user accessing Community Investment - Enterprise would have experience slow page load times and occasional page timeouts. ## Root Cause An application change intended to improve the overall performance of the application consumed significantly more resources than expected, which caused the service to be unable to effectively respond to user requests in a timely manner. ## Future Mitigation We are re-evaluating the code change to ensure that the functionality is consuming resources more efficiently. ## Timeline of Events * 10:23 MT - Service resource consumption becomes fully saturated * 12:09 MT - Internal tooling is identified as a potential source of resource contention * 12:19 MT - The identified internal tool is removed from service, improving resource contention, and restoring service * 12:30 MT - Further investigation identified an additional source of resource contention * 15:30 MT - Offending code change reverted, and internal tooling re-enabled * 18:30 MT - Deployed larger web servers