Employee Engagement (Spark), Alaya and API Client Credit Card Donations Failing

Update timeline

1. investigating Jan 16, 2025, 07:30 PM UTC

   We are currently experiencing an issue with donation processing via credit card. Users may see failure messages when attempting to make a donation payment. All other functionalities are working as expected. We are aware of the problem and are working with our payment service providers to investigate and rectify the issue.

2. identified Jan 16, 2025, 07:33 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Jan 16, 2025, 07:56 PM UTC

   All American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card donations are now working as expected. We are continuing on working through mitigation for our remaining foundations

4. identified Jan 16, 2025, 08:59 PM UTC

   We have successfully enabled credit card donation mitigation across our foundations as follows: - American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card functionality has been fully restored. - Online Giving Foundation (India) users are able to donate through payroll, if enabled in their program. - All other foundations are able to donate through PayPal. We are continuing to resolve remaining issues with Apple Pay and donations to campaigns.

5. monitoring Jan 16, 2025, 09:57 PM UTC

   We have successfully enabled credit card donation mitigation across our foundations as follows: - American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card functionality has been fully restored. - Online Giving Foundation (India) users are able to donate through payroll, if enabled in their program. - All other foundations are able to donate through PayPal.

6. monitoring Jan 16, 2025, 11:50 PM UTC

   - We have discovered issues with credit card processing through our Community Impact (CI) Portal and the inability to set up a recurring credit card donation in Employee Engagement (Spark). - We are investigating temporary solutions while we work with our payment processing partner to implement a permanent fix. - American Online Giving Foundation (AOGF) and Canadian Online Foundation (COGF) Credit Card functionality has been fully restored. - Online Giving Foundation (India) users are able to donate through payroll, if enabled in their program. - All other foundations are able to donate through PayPal.

7. monitoring Jan 17, 2025, 10:49 PM UTC

   - Credit card functionality for AOGF, India and Singapore have been fully restored. - The majority of COGF credit card functionality has been restored. - We continue to monitor credit card and Apple Pay performance for ongoing stability. - All other credit card flows through other Foundations are able to donate through PayPal.

8. identified Jan 20, 2025, 07:42 PM UTC

   - All credit card flows(including via Apple pay) have been fully restored and complete functionality is available across our platforms. - We continue to monitor performance across all platforms.

9. identified Jan 20, 2025, 09:57 PM UTC

   - All credit card flows (including via Apple pay) have been restored but we are experiencing slightly longer processing times between credit card donation submission and the success pages across our platforms. - We continue to monitor performance closely.

10. monitoring Jan 21, 2025, 12:35 AM UTC

    - All credit card flows (including via Apple pay) have been restored but we are experiencing slightly longer processing times between credit card donation submission and the success pages across our platforms. - We continue to monitor performance closely.

11. monitoring Jan 23, 2025, 08:26 PM UTC

    - All credit card flows (including via Apple pay) have been restored but we are experiencing slightly longer processing times between credit card donation submission and the success pages across our platforms. - We are actively monitoring the issue and working towards resolving the delay. Thank you for your continued patience.

12. monitoring Jan 27, 2025, 06:28 PM UTC

    All credit card flows (including via Apple pay) have been restored and normal processing times have resumed. We are monitoring performance.

13. resolved Jan 28, 2025, 03:44 PM UTC

    All credit card flows (including via Apple pay) have been restored and normal processing times have resumed.

14. postmortem Mar 03, 2025, 04:44 AM UTC

    ### Summary On January 16, a key payment processing partner experienced a critical infrastructure failure, leading to widespread disruptions in credit card and Apple Pay donation processing across Spark, Alaya, API, and the CI-Portal. While alternative payment methods such as MyFunds, PayPal, and Payroll remained available, certain regions or client programs without these options faced a complete inability to process donations. Immediate mitigations were implemented, including a failover to an alternative payment processing partner, which immediately restored functionality for almost all users. By January 17, all donation flows, processed through the American Online Giving Foundation \(AOGF\) and in India and Singapore, were restored. However, CI-Portal and specific donation flows remained impacted. Full restoration of donation processing was achieved by January 20, with some delays in processing times, which returned to normal by January 26. Throughout the incident, there was no data loss. To prevent similar incidents in the future, we are strengthening our partnerships with payment processors to improve system resilience, accelerating the implementation of a comprehensive failover process for all donation flows, and exploring additional payment processing partnerships to provide greater redundancy. ### Impact Donation processing via credit card and Apple Pay was unavailable for all users of Spark, Alaya, API, and CI-Portal. Affected users encountered a generic browser error when attempting to make donations, as the credit card form failed to load. Where alternative payment methods such as MyFunds, PayPal, or Payroll were available, users could continue donating. However, in countries or client programs where these options were not accessible, no viable donation path existed. Within the first thirty minutes of the incident, users donating through American Online Giving Foundation \(AOGF\) and Canadian Online Giving Foundation \(COGF\) were able to resume credit card donations. However, CI-Portal donations and less-common donation flows remained affected. After a further hour and thirty minutes PayPal was available across all sites, in regions it is supported. By 16:42 MT on January 17, donations through AOGF, and in India and Singapore, were possible but experienced slower processing times. By January 20 at 12:00 MT, all donation flows were restored, albeit with continued slow processing. Processing times returned to normal by January 26. There was no impact on data integrity. ### Root Cause * A key payment processing partner experienced a catastrophic infrastructure failure, requiring extensive remediation. * The failure was due to connectivity-related issues. * A failover to an alternative payment processor was successfully executed, restoring the majority of donation flows. * Some donation flows remained impacted as they were still in the process of being integrated into the comprehensive failover process. ### Future Mitigation * We will be working with our current payment processing vendors to improve resilience. * We are continuing our implementation of the comprehensive fail-over process to all donation flows. * We will be sourcing additional partners so we have further defence in payment processing. ### Timeline of Events Thu 16 Jan * 11:42MT - Payment processing error alert triggered. * 12:20MT - Failover to alternative payment processing partner complete. \(First major mitigation.\) * 13:37MT - PayPal enabled for all sites. \(Second Major Mitigation\) * 14:38MT - ApplePay Disabled. \(Third Major Mitigation\) * 14:39MT - Incident now considered mitigated, with some reduced donation capability in some regions. Fri 17 Jan * 13:45MT - Indication that there are some successful API donations occurring via our original partner. * 16:42MT - AOGF, India, Singapore re-enabled fully. Mon 20 Jan * 12:00MT - All donation flows processing successfully.