Benevity Outage History

Started Sep 18, 2025, 05:52 PM UTC· resolved Sep 19, 2025, 11:13 PM UTC

1. identified Sep 18, 2025, 05:52 PM UTC

   We are currently investigating an issue where our system's scheduled services have triggered duplicate actions for some users, such as sending duplicate emails. This issue may affect all clients. Our team has identified the cause and is actively working on a resolution. We apologize for any inconvenience this may cause and appreciate your patience. We will provide an update as soon as the fix is deployed and validated.

2. monitoring Sep 18, 2025, 06:17 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 19, 2025, 11:13 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 17, 2025, 06:05 PM UTC· resolved Sep 18, 2025, 05:17 PM UTC

1. identified Sep 17, 2025, 06:05 PM UTC

   We are currently experiencing an issue affecting the Versaic platform where the 'Get Started' button may be missing and Cause Details are not displaying correctly. Our team has successfully identified the root cause of this problem and is actively working on implementing a fix. We are in the process of preparing and deploying this solution to our production environment.

2. monitoring Sep 17, 2025, 11:09 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 18, 2025, 05:17 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 07, 2025, 08:09 PM UTC· resolved Aug 08, 2025, 05:06 AM UTC

1. investigating Aug 07, 2025, 08:09 PM UTC

   We are currently investigating an issue where client users cannot create a new contact on the organization profile or change proposal submitter.

2. identified Aug 07, 2025, 08:52 PM UTC

   We've identified the root of the issue and our team is actively implementing a solution to restore full functionality. We appreciate your patience while we work on this. More updates will follow as they become available.

3. resolved Aug 29, 2025, 09:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 17, 2025, 02:00 AM UTC· resolved Jul 17, 2025, 09:00 AM UTC

1. investigating Jul 17, 2025, 03:18 PM UTC

   We are currently investigating the issue. Some applications such as Budget Module, Proposal Collaboration, Mass actions are impacted.

2. resolved Jul 17, 2025, 03:19 PM UTC

   This incident has been resolved.

3. postmortem Aug 01, 2025, 08:25 PM UTC

   ### Summary At 2:00 a.m. UTC on July 17, 2025, a partial service impairment occurred in the Versaic platform, impacting Payment Management, Background Tasks, and Proposal Collaboration features for clients using the third-party Budget Module. The issue was identified through internal monitoring, and mitigation efforts began immediately. A resolution was implemented, and full system functionality was restored by 9:00 a.m. UTC on the same day. ### Impact Between 2:00 a.m. and 9:00 a.m. UTC on July 17, 2025, clients using the third-party Budget Module in Versaic may have encountered errors and inoperable functionality when accessing Payment Management or Proposal Collaboration workflows. As part of the recovery process, the engineering team reprocessed all background tasks and jobs affected during the outage to ensure full and accurate system restoration. ### Root Cause The incident was caused by a configuration update that unintentionally disrupted the Versaic system’s access to its database. The changes, deployed independently of the application code, were mistakenly applied across all environments due to a bug in the configuration script. ### Future Mitigation 1. The Versaic configuration method will be updated to align with code deployment. 2. The CI/CD pipeline will be updated to ensure a connectivity test after each configuration update for role-based operability. 3. Maintain, track and deploy all configuration settings in code \(via Terraform\) for greatest environment isolation. ### Timeline of Events * July 16, 2025 \(4 PM UTC\): Configuration change was created. * July 17, 2025 \(2 AM UTC\): The system picked up the new configuration, issues begin. * July 17, 2025 \(9 AM UTC\): The configuration was reverted.

Read the full incident report →

Started Jul 10, 2025, 06:50 PM UTC· resolved Jul 10, 2025, 07:09 PM UTC

1. investigating Jul 10, 2025, 06:50 PM UTC

   We are currently investigating an issue where applicants are unable to submit forms in Community Investment Essential (formerly Benevity Grants).

2. monitoring Jul 10, 2025, 06:58 PM UTC

   We have taken action to correct the issue; functionality has been restored and we are monitoring for any further issues or reoccurrences

3. resolved Jul 10, 2025, 07:09 PM UTC

   This incident has been resolved.

4. postmortem Aug 01, 2025, 08:11 PM UTC

   ## Summary On the morning of July 9, 2025, a code deployment to the Grants and Forms systems introduced an issue that began impacting clients later that day. Specifically, users encountered errors when attempting to submit grant applications. By July 10, 2025, the issue had escalated, prompting an internal investigation that identified the root cause as unresolved endpoints and invalid nodes within the Forms system. A formal incident was declared at 12:26 p.m. MT on July 10. Immediate corrective actions were taken, including addressing the affected endpoints and resetting the Forms system. These efforts successfully restored full functionality by 12:54 p.m. MT. ## Impact By the afternoon of July 9, 2025, some users began experiencing errors when attempting to submit grant applications through the Forms system. This issue continued into July 10, temporarily preventing certain organizations from successfully completing their grant submissions. ## Root Cause The incident originated from a deployment issue that revealed an unaddressed edge case in the service discovery process. This led to an incorrect configuration in the service map, causing the Forms system to route traffic to non-functional components of the Grants application. As a result, clients encountered submission errors when attempting to complete grant applications. ## Future Mitigation To help prevent similar issues in the future, health checks will be implemented for service routing components to ensure only healthy endpoints are used. Alerting will also be improved to more quickly detect and flag any communication failures between the Forms and Grants systems. Additionally, internal coordination processes will be enhanced to ensure faster incident resolution by the appropriate teams and subject matter experts. ## Timeline of Events * July 9, 2025, 8:24 am MT: Production deployment completed * July 9, 2025, 2:44 pm MT: Initial failures reported * July 10, 2025, 12:47 pm MT: Mitigation implemented * July 10, 2025, 12:52 pm MT: Full remediation completed * July 10, 2025, 12:54 pm MT: Incident resolved; systems fully operational

Read the full incident report →

Started Jun 23, 2025, 03:17 PM UTC· resolved Jun 23, 2025, 05:16 PM UTC

1. monitoring Jun 23, 2025, 03:17 PM UTC

   A fix has been implemented and we are monitoring the results.

2. resolved Jun 23, 2025, 05:16 PM UTC

   This incident has been resolved.

3. postmortem Jun 26, 2025, 09:12 PM UTC

   ## Summary On Monday, June 23, 2025, at approximately 08:03 Mountain Time \(MT\), a disruption occurred during a routine service update to the Versaic application platform. The issue affected user access to both the Client and Admin portals, resulting in login failures and an error message display. The incident was resolved within approximately one hour, with all services fully restored by 09:05 MT. ## Impact From 08:03 MT to 09:05 MT, users attempting to log in to the Versaic Client or Admin portals were unable to access the system. Instead of reaching the application’s dashboard, users encountered a message indicating: “An unexpected server error occurred.” During this period: * No user data was lost, corrupted, or compromised. * The issue was isolated to the login functionality; the underlying data and platform services remained secure and unaffected. ## Root Cause The interruption was caused by an unexpected failure during a database update as part of a scheduled deployment. Although automated system checks are designed to detect such issues before any changes are released to the live environment, this particular failure mode was not identified in time. As a result, the new application version was released, causing the login issues experienced by users. Our technical teams were immediately engaged upon alert and began diagnosing the issue. Within approximately 20 minutes, the primary cause was identified, and remediation actions were promptly executed. Full functionality was restored by 09:05 MT. ## Future Mitigation To help prevent similar incidents in the future, we are taking the following steps: * Enhancing failure detection mechanisms within our deployment processes. * Introducing additional validation checkpoints before changes are made live. ## Timeline of Events * 08:03 MT - Initial alert received * 08:20 MT - Major contributing cause identified * 08:25 MT - Notable action taken * 09:25 MT - Incident resolved; systems fully operational

Read the full incident report →

Started Jun 17, 2025, 07:43 PM UTC· resolved Jun 18, 2025, 04:52 PM UTC

1. investigating Jun 17, 2025, 07:43 PM UTC

   A group of private non-operating organizations in the US are currently unavailable in the platform. We have identified the root cause and are working to implement a fix.

2. identified Jun 18, 2025, 01:58 PM UTC

   We have identified the root-cause of the issue and a correction is being applied. At this time, the majority of impacted private non-operating organizations are available in the platform.

3. resolved Jun 18, 2025, 04:52 PM UTC

   At this time all impacted private non-operating organizations are available in the platform.

4. postmortem Jun 26, 2025, 09:51 PM UTC

   ## Summary On Tuesday, June 17, 2025, at 4:00 AM MDT, an unintended system behavior resulted in the temporary deactivation of approximately 108,000 U.S.-based “Private Non-Operating” nonprofit organizations from the Benevity platform. This occurred during a routine update process using IRS-provided data. As a result, affected organizations were temporarily marked as ineligible to receive donations, rewards, or matching funds through our platform. The issue was reported at 10:35 AM MDT the same day, and by June 18 at 10:24 AM MDT, all impacted nonprofits were restored to active status. Benevity teams worked with urgency and diligence to resolve the issue and minimize downstream effects for both nonprofits and donors. We recognize the importance of uninterrupted eligibility for these organizations and have already taken steps to ensure stronger protections going forward. ## Impact Approximately 108,000 U.S. “Private Non-Operating” nonprofit organizations were temporarily deactivated between 4:00 AM MDT on June 17 and 10:24 AM MDT on June 18. During this time period: * These nonprofits were unable to receive new donations, rewards, or matching contributions while inactive. * Recurring payroll donations were not impacted and continued to flow without interruption. * End users may notice a status update \(from deactivated to active\) in their donation history for affected organizations. ## Root Cause In April 2025, Benevity deployed an enhancement to support the ingestion of a new classification of IRS nonprofit data, including “Private Non-Operating” organizations, adding roughly 125,000 new organizations to the platform. However, during the June 17 IRS data update, the system inadvertently used a legacy configuration that did not include support for this new classification. As a result, these organizations were incorrectly flagged as absent from the official data and automatically deactivated in our ecosystem. Upon identifying the configuration discrepancy, our team reprocessed the IRS data using the correct, up-to-date configuration. This fully restored eligibility status for all affected organizations. ## Future Mitigation To reduce the likelihood of recurrence and reinforce data integrity, Benevity is taking the following steps: * Removal of outdated configurations from the system to prevent accidental use in future IRS data updates. * Enhanced anomaly detection: Revisiting thresholds and alerting logic to detect and flag unusually large volumes of deactivations or data changes during synchronization. * Automated safeguards: Strengthening pre-processing validation rules for third-party data ingestion, with a focus on nonprofit classification changes. ## Timeline of Events June 17, 2025 * 4:00am MDT - System deactivation begins * 10:35am MDT - Deactivations are detected * 11:05am MDT - Investigation and remediation efforts begin Jun 18, 2025 * 10:24am MDT - Restoration of all deactivated organizations completed * 2:42pm MDT - Incident resolved

Read the full incident report →

Started Jun 17, 2025, 03:08 PM UTC· resolved Jun 17, 2025, 03:08 PM UTC

1. resolved Jun 17, 2025, 03:08 PM UTC

   Affected Product: Reporting Studio for Employee Engagement and Grants. On June 12, users experienced issues with Reporting Studio dashboards failing to load, timing out, or displaying incomplete data. This was due to a wider disruption affecting our vendor for the cloud data platform and business intelligence tool, which powers Reporting Studio. What happened: The vendor experienced a temporary service interruption that impacted data visualization and scheduled report delivery within Reporting Studio. Current status: As of 12:45 PM PT on the same day, the issue was resolved. Reporting Studio dashboards and filters are loading and functioning normally since then. Action recommended: If you rely on scheduled report deliveries, you may have received an error mail about your report delivery being disrupted. We recommend manually re-running any critical reports that may have been delayed or missed during the incident window.

Read the full incident report →

Started Jun 11, 2025, 06:56 PM UTC· resolved Jun 12, 2025, 01:56 PM UTC

1. investigating Jun 11, 2025, 06:56 PM UTC

   Bulk importing of Volunteer Opportunities and Volunteer Sign-ups via CSV is currently unavailable due to a vendor incident. Users will encounter a "Unsupported file type" error when uploading a CSV. We are waiting for Flatfile to resolve their incident.

2. identified Jun 11, 2025, 06:56 PM UTC

   Bulk importing of Volunteer Opportunities and Volunteer Sign-ups via CSV is currently unavailable due to a vendor incident. Users will encounter a "Unsupported file type" error when uploading a CSV. We are waiting for Flatfile to resolve their incident.

3. monitoring Jun 11, 2025, 09:32 PM UTC

   The vendor has released a fix and we are monitoring the results.

4. resolved Jun 12, 2025, 01:56 PM UTC

   The vendor has marked this incident as resolved. All Spark functionality relating to bulk importing of Volunteer Opportunities and Volunteer Sign-ups should now be working as normal. Thank you!

Read the full incident report →

Started Apr 02, 2025, 04:23 PM UTC· resolved Apr 04, 2025, 01:11 PM UTC

1. identified Apr 02, 2025, 04:23 PM UTC

   We’re currently experiencing a delay in disbursing USD donations made through the UK Online Giving Foundation (UKOGF) during February and donations set to be disbursed at the end of March. The issue stems from a processing delay with our banking partner. We’ve escalated the matter to our banking partner and will continue to share updates here as we work towards a resolution in partnership with them.

2. resolved Apr 04, 2025, 01:11 PM UTC

   Delayed USD funds have now been released.

Read the full incident report →

Started Mar 28, 2025, 10:01 PM UTC· resolved Mar 28, 2025, 10:18 PM UTC

1. identified Mar 28, 2025, 10:01 PM UTC

   We have identified an issue causing an error when attempting to load the Dashboard of Spark Client sites. Our teams are working on a fix for this issue. At this time, only the Dashboard is affected - Donate, Search, and Volunteering functionality are still fully operational.

2. monitoring Mar 28, 2025, 10:14 PM UTC

   We have deployed a fix for the issue - Dashboard functionality has been restored to all Spark client sites. At this time all Spark sites and workflows are fully operational. Benevity's teams are monitoring for any further issues or recurrences.

3. resolved Mar 28, 2025, 10:18 PM UTC

   This incident has been resolved.

4. postmortem May 05, 2025, 01:35 AM UTC

   ### Summary On March 28, the Dashboard page on all Spark Sites failed to load due to a misconfigured release. While all other site functionality remained unaffected, users encountered an error when accessing the Dashboard. The issue was caused by an incorrect version of the Dashboard page being referenced during the release process. The version specified did not exist in production, preventing the page from rendering. The incident was identified and addressed quickly, with a corrected build deployed to production within approximately 15 minutes. The incident was fully resolved within 20 minutes of detection. To prevent recurrence, we are enhancing our build pipeline to eliminate manual versioning errors, improving the release process to support gradual rollouts, and introducing both automated and manual verification steps for future releases. ### Impact * The dashboard on all Spark Sites failed to load, displaying an error. * All other site functionality remained fully operational. ### Root Cause A release of the new Dashboard page was manually incorrectly configured to reference a version unavailable in production. ### Future Mitigation * Improve build pipeline to eliminate manual versioning errors. * Implement gradual rollout processes for upgraded pages. * Add additional verification for pull requests triggering production releases. * Require manual verification of releases. * Deploy automated alerting for failed rendering of upgraded pages in production. ### Timeline of Events * 2025/03/28 - 15:48MT: Identification of issue in production * 2025/03/28 - 15:50MT: Incident triggered * 2025/03/28 - 15:50MT: New build triggered * 2025/03/28 - 16:05MT: New build released to production * 2025/03/28 - 16:10MT: Incident resolved in production

Read the full incident report →

Started Mar 28, 2025, 04:33 PM UTC· resolved Mar 28, 2025, 10:27 PM UTC

1. investigating Mar 28, 2025, 04:33 PM UTC

   Versaic application is slow in production. We are currently investigating the issue. Our team is working diligently to resolve the issue as quickly as possible. We will continue to provide updates here, so please subscribe to this page for the latest information.

2. monitoring Mar 28, 2025, 05:25 PM UTC

   The team has performed steps to address the performance issue of the Versaic application, and we are monitoring to ensure stability. If you continue to experience issues, please submit a support request. Thank you for your understanding.

3. resolved Mar 28, 2025, 10:27 PM UTC

   The issue has been fully resolved, and our team is taking necessary steps to prevent this from happening again in the future. We apologize for any inconvenience this may have caused.

4. postmortem May 05, 2025, 01:38 AM UTC

   ## Summary On March 28, 2025, from 09:30 to 10:30 Versaic users experienced slower than usual performance from the Versaic service. ## Impact Some users accessing the service during that period received very slow page load times. ## Root Cause Routine monitoring of the Versaic service alerted to required system maintenance that was then actioned. That maintenance process consumed significantly more resources than expected, which reduced the system’s capacity to respond to user requests. ## Future Mitigation This maintenance process typically does not cause the level of impact that we saw during this incident, but as a preventative measure, we will move that process to lower traffic times to reduce the likelihood of impacting user experience. ## Timeline of Events * 09:00 MT - System maintenance initiated * 09:30 MT - User experience degradation identified * 10:21 MT - System maintenance process suspended * 10:30 MT - User experience degradation resolved

Read the full incident report →

Started Mar 27, 2025, 06:44 PM UTC· resolved Mar 27, 2025, 10:07 PM UTC

1. identified Mar 27, 2025, 06:44 PM UTC

   We have identified an issue on Versaic affecting some organizations where they are showing as 'Not Disbursable'. Our team is actively working on a fix, which includes both a system update and correcting the data for affected causes. We will keep you posted on the progress and next steps shortly. Thanks for your patience!

2. monitoring Mar 27, 2025, 09:46 PM UTC

   The issue has been resolved, and we are monitoring to ensure stability. Thank you for your understanding.

3. resolved Mar 27, 2025, 10:07 PM UTC

   The issue has been fully resolved, and our team has verified that the cause disbursable status for the affected clients is resolved. Our team has taken all necessary steps to prevent this from happening again in the future. We apologize for any inconvenience this may have caused.

4. postmortem May 05, 2025, 01:41 AM UTC

   ## Summary On March 27th, 2025 a small number of Versaic clients reported that a significant number of eligible causes were showing up as 'Not Disbursable' despite confirming that their profiles and banking information were up to date. ## Impact This issue was impacting multiple organizations across UK, LATAM, and APAC regions and was delaying the grant disbursement for the affected clients. ## Root Cause An update to an API that determine’s a cause’s disbursable status was the root cause of the issue. ## Future Mitigation The Versaic team will collaborate with the team responsible for the API to get the cause details on how to elegantly handle both USD and international causes. ## Timeline of Events * 11:00 MT - Incident reported. * 11:22 MT - Root cause identified and a fix in progress. * 3:00 MT - Deployment of the fix completed; Additional script executed to update all affected causes * 3:15 MT - Verified that the previously affected causes are correctly updated in Versaic. Incident resolved.

Read the full incident report →

Started Mar 25, 2025, 06:27 PM UTC· resolved Mar 25, 2025, 08:07 PM UTC

1. investigating Mar 25, 2025, 06:27 PM UTC

   We are experiencing an issue with Login for the following applications: Grants, Reporting Studio, Employee Groups, API, and some Mobile Clients. We are investigating.

2. identified Mar 25, 2025, 06:34 PM UTC

   We are experiencing an issue with Login for the following applications: Grants, Reporting Studio, Employee Groups, API, and a few Mobile Clients. One of our 3rd party providers has identified an issue, and is working to remediate.

3. identified Mar 25, 2025, 06:56 PM UTC

   Our 3rd party provider has identified the issue, and is currently rolling out the solution. We are starting to see some recovery in the login flow for some clients.

4. monitoring Mar 25, 2025, 07:34 PM UTC

   Our 3rd party provider has identified the issue, and has rolled out the solution. Login issues seem to have recovered. We are monitoring.

5. resolved Mar 25, 2025, 08:07 PM UTC

   This incident has been resolved.

6. postmortem May 05, 2025, 01:45 AM UTC

   ## Summary Benevity’s external identity provider had an internal outage in their cloud infrastructure. As a result, users were unable to login if the product and client used our external identity provider for authentication. The impact started on March 25th at 12.09 PM MDT, and was mitigated on March 25th at 12.57 PM MDT. ## Impact Users were unable to log in if the product and client used our external identity provider for authentication. The impact started on March 25th at 12.09 PM MDT, and was mitigated on March 25th at 12.57 PM MDT. ## Root Cause Our external identity provider had an internal outage in their cloud infrastructure. As a result, users were unable to log in if the product and client used our external identity provider for authentication. ## Future Mitigation * Improve proactive monitoring and alerts. * Consider implementing graceful degradation with clear user messaging. * Enhance login error messages for better user guidance. ## Timeline of Events * 12:09 PM MT - Initial alert received * 12.19 PM MT - Identified issues with our external identity provider * 12:57 PM MT - Incident resolved; systems fully operational

Read the full incident report →

Started Mar 20, 2025, 01:01 AM UTC· resolved Mar 20, 2025, 02:07 AM UTC

1. investigating Mar 20, 2025, 01:01 AM UTC

   We have received reports of clients receiving 504 bad gateway errors. Teams are currently investigating the cause and working on a resolution.

2. monitoring Mar 20, 2025, 01:24 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 20, 2025, 02:07 AM UTC

   This incident has been resolved.

4. postmortem Apr 21, 2025, 03:38 PM UTC

   ## **Summary** On March 19, 2025, from 6:30pm MT to 7:15pm MT, and on March 20 from 10:20am MT to 10:30am MT, Spark client users would have experienced significant degradation when attempting to login to their Spark site and would likely have experienced error messages when attempting to perform any operations. Significant performance issues related to resource contention on the platform instances created a situation where client requests to the platform were unable to complete and timed out, resulting in an inability for users to successfully perform any actions or access any site pages. ## **Impact** For a period of 45 minutes on March 19, 2025, and a period of 10 minutes on March 20, 2025, all users attempting to access their Spark site would have either encountered an error message when attempting to login, or experiencing significant degradation when trying to access any pages within the site. ## **Root Cause** While Benevity’s clients experienced an outage/degradation of their Spark client sites, investigation determined the root cause originated from the Benevity Platform, the backend API service used by Spark. Within the Benevity Platform codebase, the team identified a non-performant section of code that was related to the generation of donation receipt PDFs. In most normal circumstances, this section of code operated well within reasonable expectations of response time and returned results within seconds. However, in certain circumstances and with specific data sets, this section of code went into a significantly degraded state of performance and resulted in additional downstream impacts to subsequent requests on the platform from Spark client sites. Because this was not just one single large and slow query to the database, this took longer for the technical team to narrow down the specific section of code. Once the problematic section of code was identified, the team was able to implement a fix to reduce the complexity of that particular section of code and substantially improve its performance at the same time. ## **Future Mitigation** * The complex and inefficient piece of code was updated and substantially improved the overall performance of receipt generation for all users on the platform. * Additional improvements to non-performant areas of the codebase are being scoped out as part of Benevity’s Quality Assurance program. ## **Timeline of Events** March 19, 2025 * 6:30pm MT - Initial alert received * 6:56pm MT - Resource contention in platform identified * 7:15pm MT - Platform instances restarted * 7:15pm MT - Spark fully operational March 20, 2025 * 10:20am MT - Initial alert received * 10:30am MT - Platform instances restarted * 10:30am MT - Spark fully operational with mitigations in place March 27, 2025 * 1:16pm MT - Issue reproduced in non-production environment * 3:30pm MT - Team begins working on the code fix March 28, 2025 * 10:32am MT - Code fix completed March 31, 2025 * 8:52am MT - Code fix validated in non-production environment April 1, 2025 * 7:49am MT - Code fix successfully deployed to production * 8:04am MT - Code fix successfully validated in production * 8:05am MT - Remediations have been completed, all systems are operational

Read the full incident report →

Started Mar 06, 2025, 12:23 AM UTC· resolved Mar 06, 2025, 12:33 AM UTC

1. identified Mar 06, 2025, 12:23 AM UTC

   Brief outage occurred with services.

2. identified Mar 06, 2025, 12:27 AM UTC

   Systems are operational again we are continuing to investigate

3. resolved Mar 06, 2025, 12:33 AM UTC

   This incident has been resolved

4. postmortem Mar 31, 2025, 11:39 PM UTC

   ## Summary On March 5, 2025, the Benevity Donate and Volunteer service was unavailable for approximately 5 minutes starting at 17:15 MT. During this time some users accessing the Donate and Volunteer service were presented with an error screen. ## Impact The incident lasted for 5 minutes. During the outage, some users were presented with an error screen. ## Root Cause As part of improving Benevity’s security posture, modifications were made to the configuration of the load balancer of a service running at the edge of the Benevity network. Those configuration changes caused the service to recycle its availability to the load balancer. The process of deregistering and re-registering against the load balancer takes approximately 5 minutes. Once the service was re-registered, access to Donate and Volunteer was restored. When reviewing the change prior to applying it, it was not clear to the team that the change would trigger a recycle process in the vendor product. ## Future Mitigation We have disseminated the behaviour of this particular service, so that the team is more aware of this specific behaviour. This change was validated in our non-production environments, but was not observed by the team applying the change in those environments. We will be implementing some additional monitoring in our non-production environments to alert on this kind of failure in the future. ## Timeline of Events * 17:15 MT - Configuration team applied * 17:15 MT - Alerting indicates failure conditions * 17:20 MT - Service recycle process completed, and service restored

Read the full incident report →

Started Mar 03, 2025, 07:11 PM UTC· resolved Mar 04, 2025, 07:05 PM UTC

1. identified Mar 03, 2025, 07:11 PM UTC

   We have identified that we are are seeing a slower than normal response from the Versaic UI. We are investigating the cause and looking at remediation steps.

2. identified Mar 03, 2025, 07:41 PM UTC

   We have implemented a workaround, and are seeing improvement in response within the Versaic application. We are working on a permanent fix.

3. identified Mar 03, 2025, 08:58 PM UTC

   We are currently running our build pipeline to deploy a hotfix for this issue. We will post again when our deployment pipeline is complete and we have verified our fix.

4. identified Mar 03, 2025, 11:17 PM UTC

   We have deployed a fix, and we are currently monitoring and reviewing system performance.

5. monitoring Mar 04, 2025, 01:40 AM UTC

   A workaround has been deployed that appears to have alleviated the problem. We are continuing to monitor.

6. resolved Mar 04, 2025, 07:05 PM UTC

   This incident has been resolved.

7. postmortem Apr 01, 2025, 12:10 AM UTC

   ## Summary On March 3, 2025 from 10:23 MT to 12:19 MT, users accessing Community Investment - Enterprise experienced slow page load times and occasional page timeouts as a result of application changes introduced that day. ## Impact Between 10:23 MT and 12:19 MT any user accessing Community Investment - Enterprise would have experience slow page load times and occasional page timeouts. ## Root Cause An application change intended to improve the overall performance of the application consumed significantly more resources than expected, which caused the service to be unable to effectively respond to user requests in a timely manner. ## Future Mitigation We are re-evaluating the code change to ensure that the functionality is consuming resources more efficiently. ## Timeline of Events * 10:23 MT - Service resource consumption becomes fully saturated * 12:09 MT - Internal tooling is identified as a potential source of resource contention * 12:19 MT - The identified internal tool is removed from service, improving resource contention, and restoring service * 12:30 MT - Further investigation identified an additional source of resource contention * 15:30 MT - Offending code change reverted, and internal tooling re-enabled * 18:30 MT - Deployed larger web servers

Read the full incident report →

Started Feb 27, 2025, 11:33 PM UTC· resolved Feb 28, 2025, 07:57 PM UTC

1. investigating Feb 27, 2025, 11:33 PM UTC

   Benevity Causes and Nonprofits Portal are currently experiencing degraded page load times and intermittent unavailability. We are currently Investigating.

2. monitoring Feb 28, 2025, 01:41 AM UTC

   Benevity Causes and Nonprofits Portal is available and we are seeing normal page load times. We are continuing to monitor for any further issues or recurrences.

3. resolved Feb 28, 2025, 07:57 PM UTC

   This incident has been resolved.

4. postmortem May 05, 2025, 01:51 AM UTC

   ### **Summary** At approx. 4:00pm MST on Thursday, Feb. 27th the Causes Portal site was reported to be returning 504 error codes, somewhat sporadically. User's receiving the 504 error codes were unable to access the Cause Portal application, however, reload attempts would occasionally succeed. Investigation ultimately revealed that a recent security configuration change had resulted in the load-balancers not being able to successfully complete their health checks on the Cause Portal application instances, eventually resulting in cycling of the instances resulting in sporadic 504 errors to be returned. At approx. 5:54pm on Feb 27th the issue was resolved. ### **Impact** Cause Portal users would sporadically receive 504 errors when attempting to access the Cause Portal application, preventing or at least delaying them from completing their tasks. ### **Root Cause** A newly configured anti-host-spoofing feature prevented the load-balancers from completing their health-checks on the application instances, resulting in frequent cycling of the instances, which eventually resulted in sporadic 504 errors being returned. ### **Future Mitigation** Future configuration changes of this nature will be reviewed by the appropriate internal infrastructure team for possible negative impacts prior to approval. ### **Timeline of Events** At approx. 4:00pm MST on Thursday, Feb. 27th the Causes Portal site was reported to be returning 504 error codes, somewhat sporadically. User's receiving the 504 error codes were unable to access the Cause Portal application, however, reload attempts would occasionally succeed. Investigation ultimately revealed that a recent security configuration change had resulted in the load-balancers not being able to successfully complete their health checks on the Cause Portal application instances, eventually resulting in cycling of the instances resulting in sporadic 504 errors to be returned. At approx. 5:54pm on Feb 27th the issue was resolved.

Read the full incident report →

Started Feb 24, 2025, 09:18 PM UTC· resolved Feb 28, 2025, 06:03 AM UTC

1. investigating Feb 24, 2025, 09:18 PM UTC

   We are currently experiencing degraded page load times and intermittent unavailability of CI Essential (formerly Benevity Grants). Benevity's Engineering and DevOps teams are investigating.

2. monitoring Feb 24, 2025, 09:35 PM UTC

   All Community Investment - Essential Client sites are available and we are now seeing normal page load times. We are continuing to monitor for any further issues or recurrences.

3. resolved Feb 28, 2025, 06:03 AM UTC

   This incident has been resolved.

4. postmortem Mar 31, 2025, 11:43 PM UTC

   ### Summary On February 24, 2025, the Community Investment Essentials \(Benevity Grants\) platform experienced intermittent service disruptions starting at 3:14 PM EST. The issue was fully resolved, and normal service operation was restored by 4:30 PM EST. ### Impact During this period, some clients experienced intermittent access to the Community Investment Essentials platform. ### Root Cause The service disruption was caused by an overload of the unique submission validation check for applications. During a period of increased activity, this validation process became overwhelmed, leading to temporary service disruptions. ### Future Mitigation To prevent further disruptions, we optimized the application submission validation process to handle higher volumes of activity. This improvements will ensure greater stability and reliability, even during peak usage periods. ### Timeline of Events * Feb 24, 2025 at 3:14 PM EST - Initial alert received * Feb 24, 2025 at 3:49 PM EST - Issue identified * Feb 24, 2025 at 4:04 PM EST - Mitigation process started * Feb 24, 2025 at 4:30 PM EST - Incident resolved

Read the full incident report →

Started Feb 13, 2025, 08:30 AM UTC· resolved Feb 13, 2025, 08:30 AM UTC

1. resolved Feb 13, 2025, 09:08 PM UTC

   From 13:33 to 13:36 MT, some users experienced an error message when attempting to access Donate and Volunteer services. The incident was resolved by automated processes that detected the failing services and service was quickly restored.

Read the full incident report →

Started Jan 27, 2025, 10:34 PM UTC· resolved Jan 28, 2025, 07:08 PM UTC

1. investigating Jan 27, 2025, 10:34 PM UTC

   We have received reports of some clients receiving 500 errors when using Volunteering Search. These errors are a result of issues with Geolocation services. Teams are currently investigating cause and working on a resolution.

2. identified Jan 27, 2025, 11:19 PM UTC

   We have identified an issue that had resulted in a larger than normal number of API calls our geolocation service. For some users, this will result in a 500 error while search for Volunteer Opportunities. We have a fix in place that we expect will resolve this issue. We expect to have the fix released in the next release cycle.

3. monitoring Jan 28, 2025, 02:41 PM UTC

   A fix has been implemented and we are monitoring results. Volunteer Opportunity searches are now returning with valid results instead of showing 500 errors.

4. resolved Jan 28, 2025, 07:08 PM UTC

   Continued monitoring of logs is showing that the error is no longer present. We can safely say that this issue is now resolved.

5. postmortem Mar 03, 2025, 05:08 AM UTC

   ### Summary A temporary issue with our geolocation service caused some users to experience difficulties. This was due to an unexpectedly large number of requests, which overloaded the geolocation service. We have taken steps to resolve this issue and prevent it from happening again by changing how we store user location and when we request this information and send it to our geolocation service provider. ### Impact Some users recently experienced issues searching for volunteer opportunities. Our search feature uses location data to find nearby opportunities. A recent update caused the system to make a large number of requests to our geoocation service, which unfortunately led to a temporary overload. This meant some users encountered an error message while searching, while others were able to search successfully. ### Root Cause A temporary issue with our geolocation service caused some users to experience difficulties. This was due to an unexpectedly large number of requests, which overloaded the geolocation service. ### Future Mitigation To prevent this issue from recurring, we're making some improvements to how we handle location information. Going forward, the system will only store a user's full address when they specifically update it. This allows us to still display opportunities that are closest to the user.

Read the full incident report →

Started Jan 16, 2025, 07:30 PM UTC· resolved Jan 28, 2025, 03:44 PM UTC

1. investigating Jan 16, 2025, 07:30 PM UTC

   We are currently experiencing an issue with donation processing via credit card. Users may see failure messages when attempting to make a donation payment. All other functionalities are working as expected. We are aware of the problem and are working with our payment service providers to investigate and rectify the issue.

2. identified Jan 16, 2025, 07:33 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Jan 16, 2025, 07:56 PM UTC

   All American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card donations are now working as expected. We are continuing on working through mitigation for our remaining foundations

4. identified Jan 16, 2025, 08:59 PM UTC

   We have successfully enabled credit card donation mitigation across our foundations as follows: - American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card functionality has been fully restored. - Online Giving Foundation (India) users are able to donate through payroll, if enabled in their program. - All other foundations are able to donate through PayPal. We are continuing to resolve remaining issues with Apple Pay and donations to campaigns.

5. monitoring Jan 16, 2025, 09:57 PM UTC

   We have successfully enabled credit card donation mitigation across our foundations as follows: - American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card functionality has been fully restored. - Online Giving Foundation (India) users are able to donate through payroll, if enabled in their program. - All other foundations are able to donate through PayPal.

6. monitoring Jan 16, 2025, 11:50 PM UTC

   - We have discovered issues with credit card processing through our Community Impact (CI) Portal and the inability to set up a recurring credit card donation in Employee Engagement (Spark). - We are investigating temporary solutions while we work with our payment processing partner to implement a permanent fix. - American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card functionality has been fully restored. - Online Giving Foundation (India) users are able to donate through payroll, if enabled in their program. - All other foundations are able to donate through PayPal.

7. monitoring Jan 17, 2025, 10:49 PM UTC

   - Credit card functionality for AOGF, India and Singapore have been fully restored. - The majority of COGF credit card functionality has been restored. - We continue to monitor credit card and Apple Pay performance for ongoing stability. - All other credit card flows through other Foundations are able to donate through PayPal.

8. identified Jan 20, 2025, 07:42 PM UTC

   - All credit card flows(including via Apple pay) have been fully restored and complete functionality is available across our platforms. - We continue to monitor performance across all platforms.

9. identified Jan 20, 2025, 09:57 PM UTC

   - All credit card flows (including via Apple pay) have been restored but we are experiencing slightly longer processing times between credit card donation submission and the success pages across our platforms. - We continue to monitor performance closely.

10. monitoring Jan 21, 2025, 12:35 AM UTC

    - All credit card flows (including via Apple pay) have been restored but we are experiencing slightly longer processing times between credit card donation submission and the success pages across our platforms. - We continue to monitor performance closely.

11. monitoring Jan 23, 2025, 08:26 PM UTC

    - All credit card flows (including via Apple pay) have been restored but we are experiencing slightly longer processing times between credit card donation submission and the success pages across our platforms. - We are actively monitoring the issue and working towards resolving the delay. Thank you for your continued patience.

12. monitoring Jan 27, 2025, 06:28 PM UTC

    All credit card flows (including via Apple pay) have been restored and normal processing times have resumed. We are monitoring performance.

13. resolved Jan 28, 2025, 03:44 PM UTC

    All credit card flows (including via Apple pay) have been restored and normal processing times have resumed.

14. postmortem Mar 03, 2025, 04:44 AM UTC

    ### Summary On January 16, a key payment processing partner experienced a critical infrastructure failure, leading to widespread disruptions in credit card and Apple Pay donation processing across Spark, Alaya, API, and the CI-Portal. While alternative payment methods such as MyFunds, PayPal, and Payroll remained available, certain regions or client programs without these options faced a complete inability to process donations. Immediate mitigations were implemented, including a failover to an alternative payment processing partner, which immediately restored functionality for almost all users. By January 17, all donation flows, processed through the American Online Giving Foundation \(AOGF\) and in India and Singapore, were restored. However, CI-Portal and specific donation flows remained impacted. Full restoration of donation processing was achieved by January 20, with some delays in processing times, which returned to normal by January 26. Throughout the incident, there was no data loss. To prevent similar incidents in the future, we are strengthening our partnerships with payment processors to improve system resilience, accelerating the implementation of a comprehensive failover process for all donation flows, and exploring additional payment processing partnerships to provide greater redundancy. ### Impact Donation processing via credit card and Apple Pay was unavailable for all users of Spark, Alaya, API, and CI-Portal. Affected users encountered a generic browser error when attempting to make donations, as the credit card form failed to load. Where alternative payment methods such as MyFunds, PayPal, or Payroll were available, users could continue donating. However, in countries or client programs where these options were not accessible, no viable donation path existed. Within the first thirty minutes of the incident, users donating through American Online Giving Foundation \(AOGF\) and Canadian Online Giving Foundation \(COGF\) were able to resume credit card donations. However, CI-Portal donations and less-common donation flows remained affected. After a further hour and thirty minutes PayPal was available across all sites, in regions it is supported. By 16:42 MT on January 17, donations through AOGF, and in India and Singapore, were possible but experienced slower processing times. By January 20 at 12:00 MT, all donation flows were restored, albeit with continued slow processing. Processing times returned to normal by January 26. There was no impact on data integrity. ### Root Cause * A key payment processing partner experienced a catastrophic infrastructure failure, requiring extensive remediation. * The failure was due to connectivity-related issues. * A failover to an alternative payment processor was successfully executed, restoring the majority of donation flows. * Some donation flows remained impacted as they were still in the process of being integrated into the comprehensive failover process. ### Future Mitigation * We will be working with our current payment processing vendors to improve resilience. * We are continuing our implementation of the comprehensive fail-over process to all donation flows. * We will be sourcing additional partners so we have further defence in payment processing. ### Timeline of Events Thu 16 Jan * 11:42MT - Payment processing error alert triggered. * 12:20MT - Failover to alternative payment processing partner complete. \(First major mitigation.\) * 13:37MT - PayPal enabled for all sites. \(Second Major Mitigation\) * 14:38MT - ApplePay Disabled. \(Third Major Mitigation\) * 14:39MT - Incident now considered mitigated, with some reduced donation capability in some regions. Fri 17 Jan * 13:45MT - Indication that there are some successful API donations occurring via our original partner. * 16:42MT - AOGF, India, Singapore re-enabled fully. Mon 20 Jan * 12:00MT - All donation flows processing successfully.

Read the full incident report →

Started Oct 28, 2024, 07:00 PM UTC· resolved Jun 18, 2025, 02:26 PM UTC

1. identified Oct 28, 2024, 07:00 PM UTC

   On October 22, 2024, we were informed by UKOGF's FX processor, StoneX Group Inc, that they cannot send funds from UKOGF to India due to the Reserve Bank of India (RBI) needing additional information for regulatory compliance. We are actively seeking solutions to resolve the issue and have requested a meeting with in-country counsel to see what might be possible. Please note that this issue does not affect the Online Giving Foundation in India, it only affects cross-border donations going into India from UKOGF. Until the RBI, StoneX, and UKOGF reach a resolution, disbursements from UKOGF to India will have to be put on hold. This issue is impacting all of UKOGF's banking partners from sending funds to India. No further action is required from clients at this time. We appreciate your patience and will provide updates as they become available. We'll continue to provide updates here, so please subscribe to this page for the latest information.

2. identified Nov 05, 2024, 11:31 PM UTC

   Update: We continue to actively work with our banking partners to resolve this issue. We are investigating alternative payment railways to ensure donations to India via the UKOGF are processed successfully. To stay informed with the latest updates, please consider subscribing. Your ongoing patience and understanding as we navigate through this challenge are greatly appreciated.

3. identified Nov 13, 2024, 03:19 PM UTC

   We continue to actively work with our banking partners to resolve this issue. We are testing alternative payment railways to ensure donations to India via the UKOGF are processed successfully. To stay informed with the latest updates, please consider subscribing. Your ongoing patience and understanding as we navigate through this challenge are greatly appreciated.

4. identified Nov 29, 2024, 04:03 PM UTC

   We’ve identified a solution for processing USD-funded donations to Indian nonprofits and will begin releasing previously held disbursements. During this process, all funds will be sent in USD to recipient nonprofits, with a minimum disbursement threshold of $50 to cover transaction fees. Testing is still ongoing for non-USD-funded donations, and we are working diligently to resolve this as soon as possible. We appreciate your patience and will continue to keep you informed on progress.

5. identified Jan 18, 2025, 04:44 PM UTC

   We are continuing to use our alternative disbursement route for all USD-funded donations to Indian nonprofits. Testing is still underway for non-USD-funded donations, and we are working diligently to resolve the issue as soon as possible. We appreciate your patience and will keep you informed of our progress.

6. monitoring Feb 18, 2025, 04:06 PM UTC

   All previously held donations have now been released by UKOGF and are currently with an intermediary banking provider, pending transfer to the recipient non-profits. We anticipate this process to take a few days while we provide the additional information required to release these funds. We are committed to expediting this transfer and will provide another update on February 21, 2025, or sooner if there is significant news.

7. monitoring Feb 24, 2025, 04:01 PM UTC

   We are experiencing an issue with UKOGF India transactions from the recent disbursement. All transactions to nonprofits in India through UKOGF are currently being rejected by the intermediary bank. We had previously completed 35 successful test transactions, however, we are now investigating why all subsequent transactions are failing. We are working closely with our banking partners to identify the issue.

8. monitoring Mar 18, 2025, 09:25 PM UTC

   Ongoing challenges with cross-border donations to India through UKOGF continue due to the banking changes introduced by the Reserve Bank of India last year. Discussions with banking partners, advisors, and industry experts confirm these issues extend beyond Benevity and UKOGF. To continue support for Indian nonprofits, USD donations through UKOGF are now being disbursed through a fully compliant interim solution. The first disbursement utilizing this solution was executed on December 31, 2024 and the next is scheduled for March 20, 2025. No currency, other than USD, has successfully been sent to India via UKOGF since the problem was identified in October 2024. These donations are temporarily being held. We are actively testing solutions for other currencies, and expect to provide further updates in the coming weeks.

9. monitoring May 01, 2025, 12:41 PM UTC

   USD Update: Donations made in USD to Indian nonprofits through UKOGF are now being disbursed by the beginning of each month. Other Currency Update: Since the issue was identified in October 2024, no other currencies have been successfully sent to India via UKOGF. We’re completing the final testing of the new compliant disbursement solution, and—if successful—aim to release the held funds in May. We’ll share another update once testing is complete in the coming weeks.

10. monitoring May 13, 2025, 01:12 PM UTC

    We’ve successfully completed testing for all remaining currencies. Donations made by March 31, 2025, that meet the minimum threshold and have an approved FCRA (Foreign Contribution Regulation Act) account listed on the nonprofit’s cause profile, will be disbursed to Indian nonprofits via UKOGF by May 20, 2025. Following this release, all funding currencies — including USD — will be disbursed to Indian nonprofits on a monthly basis within the first few business days of each month following the primary UKOGF disbursement.

11. resolved Jun 18, 2025, 02:26 PM UTC

    This incident has been resolved.

Read the full incident report →