Applicants are Unable to Submit Forms in CI Essentials (formerly Benevity Grants)

Update timeline

1. investigating Jul 10, 2025, 06:50 PM UTC

   We are currently investigating an issue where applicants are unable to submit forms in Community Investment Essential (formerly Benevity Grants).

2. monitoring Jul 10, 2025, 06:58 PM UTC

   We have taken action to correct the issue; functionality has been restored and we are monitoring for any further issues or reoccurrences

3. resolved Jul 10, 2025, 07:09 PM UTC

   This incident has been resolved.

4. postmortem Aug 01, 2025, 08:11 PM UTC

   ## Summary On the morning of July 9, 2025, a code deployment to the Grants and Forms systems introduced an issue that began impacting clients later that day. Specifically, users encountered errors when attempting to submit grant applications. By July 10, 2025, the issue had escalated, prompting an internal investigation that identified the root cause as unresolved endpoints and invalid nodes within the Forms system. A formal incident was declared at 12:26 p.m. MT on July 10. Immediate corrective actions were taken, including addressing the affected endpoints and resetting the Forms system. These efforts successfully restored full functionality by 12:54 p.m. MT. ## Impact By the afternoon of July 9, 2025, some users began experiencing errors when attempting to submit grant applications through the Forms system. This issue continued into July 10, temporarily preventing certain organizations from successfully completing their grant submissions. ## Root Cause The incident originated from a deployment issue that revealed an unaddressed edge case in the service discovery process. This led to an incorrect configuration in the service map, causing the Forms system to route traffic to non-functional components of the Grants application. As a result, clients encountered submission errors when attempting to complete grant applications. ## Future Mitigation To help prevent similar issues in the future, health checks will be implemented for service routing components to ensure only healthy endpoints are used. Alerting will also be improved to more quickly detect and flag any communication failures between the Forms and Grants systems. Additionally, internal coordination processes will be enhanced to ensure faster incident resolution by the appropriate teams and subject matter experts. ## Timeline of Events * July 9, 2025, 8:24 am MT: Production deployment completed * July 9, 2025, 2:44 pm MT: Initial failures reported * July 10, 2025, 12:47 pm MT: Mitigation implemented * July 10, 2025, 12:52 pm MT: Full remediation completed * July 10, 2025, 12:54 pm MT: Incident resolved; systems fully operational