Intermittent Unavailability of Causes (Non-Profit) Portal

Update timeline

1. investigating Feb 27, 2025, 11:33 PM UTC

   Benevity Causes and Nonprofits Portal are currently experiencing degraded page load times and intermittent unavailability. We are currently Investigating.

2. monitoring Feb 28, 2025, 01:41 AM UTC

   Benevity Causes and Nonprofits Portal is available and we are seeing normal page load times. We are continuing to monitor for any further issues or recurrences.

3. resolved Feb 28, 2025, 07:57 PM UTC

   This incident has been resolved.

4. postmortem May 05, 2025, 01:51 AM UTC

   ### **Summary** At approx. 4:00pm MST on Thursday, Feb. 27th the Causes Portal site was reported to be returning 504 error codes, somewhat sporadically. User's receiving the 504 error codes were unable to access the Cause Portal application, however, reload attempts would occasionally succeed. Investigation ultimately revealed that a recent security configuration change had resulted in the load-balancers not being able to successfully complete their health checks on the Cause Portal application instances, eventually resulting in cycling of the instances resulting in sporadic 504 errors to be returned. At approx. 5:54pm on Feb 27th the issue was resolved. ### **Impact** Cause Portal users would sporadically receive 504 errors when attempting to access the Cause Portal application, preventing or at least delaying them from completing their tasks. ### **Root Cause** A newly configured anti-host-spoofing feature prevented the load-balancers from completing their health-checks on the application instances, resulting in frequent cycling of the instances, which eventually resulted in sporadic 504 errors being returned. ### **Future Mitigation** Future configuration changes of this nature will be reviewed by the appropriate internal infrastructure team for possible negative impacts prior to approval. ### **Timeline of Events** At approx. 4:00pm MST on Thursday, Feb. 27th the Causes Portal site was reported to be returning 504 error codes, somewhat sporadically. User's receiving the 504 error codes were unable to access the Cause Portal application, however, reload attempts would occasionally succeed. Investigation ultimately revealed that a recent security configuration change had resulted in the load-balancers not being able to successfully complete their health checks on the Cause Portal application instances, eventually resulting in cycling of the instances resulting in sporadic 504 errors to be returned. At approx. 5:54pm on Feb 27th the issue was resolved.