Benevity experienced a major incident on October 20, 2025 affecting Donate and Volunteer Core Services and Grants Essential (formerly Benevity Grants) and 1 more component, lasting 1d 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Oct 20, 2025, 11:36 AM UTC
Due to a service disruption with a key infrastructure provider all Benevity Services are in a degraded or disrupted state. Recovery is underway and services are recovering. Next update in 1 hour.
- monitoring Oct 20, 2025, 12:01 PM UTC
We are continuing to monitor for any further issues.
- monitoring Oct 20, 2025, 12:42 PM UTC
Everything seems to be working as expected but we are continuing to monitor all of our services and assessing the impact of this disruption
- monitoring Oct 20, 2025, 03:06 PM UTC
All services have been restored and are operating normally. Our teams continue to monitor the systems closely as we conduct a full assessment of the disruption's impact.
- monitoring Oct 20, 2025, 04:18 PM UTC
We are still impacted by an ongoing incident with our primary infrastructure provider. Our team is closely monitoring their status and will post updates as they become available.
- monitoring Oct 20, 2025, 05:52 PM UTC
Benevity services are recovering and are stable, and we do not expect any significant impact to end users at this time. Our team continues to monitor the situation closely and is working with the infrastructure provider to ensure full restoration.
- monitoring Oct 20, 2025, 08:34 PM UTC
Most Benevity services have recovered and are functioning as expected. We are continuing to monitor minor degradation in a few areas and are working with our infrastructure provider to ensure full restoration.
- monitoring Oct 21, 2025, 12:05 AM UTC
The majority of Benevity services have recovered and are functioning as expected. We are continuing to monitor minor degradation in a few areas and are working with our service providers to ensure full restoration.
- resolved Oct 21, 2025, 03:50 PM UTC
This incident has been resolved.
- postmortem Nov 03, 2025, 02:39 AM UTC
## Summary Between October 20, 2025, 1:11 AM MT and 4:01 PM MT, Benevity's key infrastructure provider experienced a significant outage in the US-EAST-1 region. This outage cascaded into widespread disruptions across this and other providers and Benevity services. The incident was triggered by DNS resolution failures within the key infrastructure providers systems, which subsequently caused impairments in the launch capabilities of virtual machines and Network Load Balancer health checks. These issues thereafter affected a wide range of services provided by our key infrastructure provider that Benevity services depends on. The outage affected the following key Benevity functionalities: * Payment processing * Giving and volunteering operations \(Matching Offers, Giving Opportunity and Volunteer Opportunity creation with reporting tags\) * Non-profit related systems. * User management and authentication * Search and reporting capabilities * Email notifications and communications * Various backend integrations and data processing jobs, including disbursement of funds. During the incident, our teams actively managed the situation by successfully failing over credit card donation processing to our backup system and proactively pausing donation disbursement functionality to ensure integrity. As services returned, our team managed the backlog and executed retry measures to remediate the impact. While many workflows continued functioning, some end-user experiences were affected during the outage window: * Donation attempts experienced disruptions in select cases * Some automated match requests did not receive auto-approval and required manual review * Some email communications experienced delays or failures that were subsequently remediated * Administrative functions like giving and volunteer opportunity management, and search capabilities experienced degradation. By end of business October 20th, the vast majority of systems were operational. October 21st confirmed full recovery with no outstanding issues. ## Impact During the incident, Benevity experienced service interruptions across several core systems as a result of a widespread outage from our primary infrastructure provider. The failure caused temporary throttling of system resources and delays in launching new compute environments, which in turn affected the performance and availability of key Benevity services between 00:50 MT and 22:58 MT on October 20, 2025. **Giving and Transactional Systems** were impacted early in the event, with credit card donations temporarily unavailable for approximately 90 minutes before successfully failing over to a stable backup flow. While a small number of donation attempts failed during this period, Benevity has verified that all processed transactions were reconciled and that data integrity was fully maintained. Disbursement processing was proactively paused later in the morning to ensure consistency, resulting in a one-day delay in initiating batch disbursements. **API and Administrative Tools** experienced intermittent unavailability during the same window. Clients integrating via Benevity APIs encountered elevated error rates, and administrators faced interruptions when managing Giving and Volunteering Opportunities. Temporary failures in tag management and match request automation required limited rework once systems stabilized. Benevity engaged directly with affected clients to identify and address any incomplete records or pending approvals. **Automated Data and Processing Systems** experienced delays in sending notifications, importing payroll and user data, and generating scheduled reports. All impacted processes were retried and successfully completed later in the day, with no data loss or long-term effects. **Nonprofit Systems** also saw delays in background communications and validation workflows for newly nominated and onboarded nonprofits. These queued processes automatically resumed once infrastructure services recovered. Overall, while several client-facing and back-end services were temporarily unavailable or delayed throughout the day, Benevity’s data integrity remained intact, all queued operations completed successfully once stability was restored, and all client transactions and records were validated post-recovery. ## Root Cause The widespread degradation and disruption of Benevity services stemmed from a major external failure originating with Benevity's key infrastructure provider in its primary operating region. This event was categorized as a Severity 1 \(SEV1\) incident due to its sweeping impact across multiple critical applications. The Core Technical Failure \(Infrastructure Provider\) The infrastructure provider confirmed that the incident was triggered by fundamental networking issues within their platform, specifically related to DNS resolution issues for core data service endpoints. This initial failure led to cascading problems that impacted internal subsystems responsible for managing fundamental cloud resources: * Connectivity issues immediately halted the operation of critical serverless applications and disturbed access to centralized data storage services. * As the provider attempted recovery, they were forced to throttle \(rate limit\) the launch of new virtual servers. This throttling prevented essential scheduled tasks and automated workflows from executing normally. * This outage was not isolated to Benevity; multiple critical business partners and tools relied upon for operations were also degraded by the same infrastructure failure. ## Future Mitigation Benevity views the recent service disruption as a critical learning opportunity to further reinforce our systems against external infrastructure volatility. Our future mitigation strategy centres on three key areas designed to maximize stability, data accuracy, and operational response, ensuring greater resilience for clients. We are implementing improvements to ensure core systems remain stable, even if external services they depend on experience outages. This includes but is not limited to improved operating procedures to disable capabilities, payment system hardening, matching offers graceful degradation, automated email retry, reporting tag loss prevention, vendor accountability, improvements to observability, updated post incident playbooks for service validation. ## Timeline of Events * October 20, 2025 - 00:50 MT - Benevity Incident automatically detected and alerted through monitoring systems. * October 20, 2025 - 01:11 MT - Key infrastructure provider system report increased error rates & latencies. * October 20, 2025 - 01:34 MT - Internal incident formally declared. * October 20, 2025 - 02:28 MT - Credit card donation functionality failed over to backup system. * October 20, 2025 - 02:28 MT - Major Contributing Cause confirmed as key infrastructure provider outage. * October 20, 2025 - 03:22 MT - Key Infrastructure provider reports initial mitigation and early signs of recovery * October 20, 2025 - 03:22 MT - Benevity services start reporting healthy checks, we continue to monitor. * October 20, 2025 - 10:33 MT - Donation Disbursement functionality disabled till reliable service is available. * October 20, 2025 - 14:41 MT - Most Benevity services have recovered and are functioning as expected. * October 20, 2025 - 15:29 MT - Previously failed Spark email notifications are sent. * October 20, 2025 - 17:00 MT - Key Infrastructure provider reporting all by services are resolved. * October 20, 2025 - 18:00 MT - Benevity holds monitoring status for the evening while our providers' systems heal. * October 21, 2025 - 09:24 MT - All Benevity systems are reported to have return to operational status.