Benevity experienced a major incident on December 11, 2024 affecting Donate and Volunteer Core Services, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Dec 11, 2024, 04:48 PM UTC
We are currently investigating an issue impacting a subset of clients that prevents users for those clients from accessing their Spark site. Our team is aware and is working diligently to resolve the issue as quickly as possible.
- resolved Dec 12, 2024, 05:43 PM UTC
The issue affecting a small subset of spark clients has been fully resolved, and our team has taken all necessary steps to prevent this from happening again in the future. We apologize for any inconvenience this may have caused.
- postmortem Dec 19, 2024, 12:44 AM UTC
### Summary On December 11, 2024, between 09:18 AM MT and 09:43 AM MT, some Spark clients encountered errors when attempting to log in to their Spark sites. Additionally, users who were already logged in may have experienced errors while navigating to other pages within the site. ## Impact Between 09:18 AM MT and 09:43 AM MT, users were unable to log in to Spark and received error messages upon attempting to do so. Additionally, users already logged in may have experienced degraded performance, with some site pages also returning error messages. ## Root Cause A technical issue with Spark's security settings caused an error that temporarily disrupted the service. The issue was quickly identified, and the team took steps to temporarily adjust the security settings to restore Spark's functionality. Once the fix was successfully implemented, the original security settings were reinstated. ## Future Mitigation The security configurations have been rolled out to all testing environments. ## Timeline of Events * 9:18am MST: Incident Detected * 9:41am MST: Disabled a specific security configuration for Spark sites * 9:43am MST: Error stopped. * 9:52am MST: Incident Mitigated * 2:43pm MST: Code change to fix security configuration deployed to Spark. * 2:58pm MST: Security feature re-enabled for Spark clients. * 3:00pm MST: Incident Resolved