Benevity experienced a major incident on September 26, 2025 affecting Donate and Volunteer Core Services, lasting 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 26, 2025, 03:40 PM UTC
Spark's donation form is currently down and users are experiencing errors when attempting to donate. We are investigating the issue.
- monitoring Sep 26, 2025, 03:56 PM UTC
A fix has been implemented and donation processing is working again. We are continuing to monitor for any adverse affects.
- resolved Sep 26, 2025, 04:08 PM UTC
This incident has been resolved.
- postmortem Oct 05, 2025, 09:41 PM UTC
## Summary On Friday, September 26, between 9:03 AM and 9:50 AM MT, users in Spark \(Donate and Volunteer\) encountered error messages when attempting to access the Donation Form and Donation History pages. During the same window, a related service that supports volunteering functionality was also unable to communicate with necessary backend services. ## Impact During the incident window, users were unable to access certain giving functionality within Spark, including the Donation Form and Donation History pages. This interruption prevented some users from completing or viewing donations and temporarily affected engagement with giving and volunteering activities. ## Root Cause An infrastructure update related to networking security configurations inadvertently resulted in our Payments processing service to become unreachable by the Spark application, causing end-users to experience an error page when trying to access the Donation Form. ## Future Mitigation We are reviewing our Engineering and Quality Assurance practices around infrastructure updates of this kind to prevent such incidents in the future. This includes enhancing our change management processes, improving validation and testing procedures, and strengthening peer review requirements for configuration updates. These efforts are aimed at reducing the likelihood of similar disruptions, increasing visibility into service dependencies, and improving the overall resilience of our production environment. ## Timeline of Events * Sept 26 @ 9:00am MT - Infrastructure update is made leading to the errors on the Donation Form. * Sept 26 @ 9:04am MT - Monitoring systems detect the service is down and page out to Engineering and SRE teams. * Sept 26 @ 9:45am MT - Root cause is identified. * Sept 26 @ 9:46am MT - Fix is put in place and service is restored.