Neon Fundraise experienced a critical incident on August 18, 2023, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Aug 21, 2023, 07:55 PM UTC
Neon Fundraise experienced a severe slowdown this (Friday) morning. Reports: Clients reported sites not loading, admin not loading, in general pages taking very long to load, some not loading at all. Issue: Our API was being overused/abused, receiving up to 5.5 requests per second, which essentially took down the system. Fix: Within 10 minutes of clients reporting issues, we moved to shut down the offending client's API access. Resolution: As of 10:35am Pacific (after enough time passed for the system to process the many requests), the system was loading at normal speeds. Future actions: Going forward we will explore introducing a rate limit to our API to ensure it cannot be taken down in this way.