Volunteering Search Resulting in 500 Errors for Some Clients

Update timeline

1. investigating Jan 27, 2025, 10:34 PM UTC

   We have received reports of some clients receiving 500 errors when using Volunteering Search. These errors are a result of issues with Geolocation services. Teams are currently investigating cause and working on a resolution.

2. identified Jan 27, 2025, 11:19 PM UTC

   We have identified an issue that had resulted in a larger than normal number of API calls our geolocation service. For some users, this will result in a 500 error while search for Volunteer Opportunities. We have a fix in place that we expect will resolve this issue. We expect to have the fix released in the next release cycle.

3. monitoring Jan 28, 2025, 02:41 PM UTC

   A fix has been implemented and we are monitoring results. Volunteer Opportunity searches are now returning with valid results instead of showing 500 errors.

4. resolved Jan 28, 2025, 07:08 PM UTC

   Continued monitoring of logs is showing that the error is no longer present. We can safely say that this issue is now resolved.

5. postmortem Mar 03, 2025, 05:08 AM UTC

   ### Summary A temporary issue with our geolocation service caused some users to experience difficulties. This was due to an unexpectedly large number of requests, which overloaded the geolocation service. We have taken steps to resolve this issue and prevent it from happening again by changing how we store user location and when we request this information and send it to our geolocation service provider. ### Impact Some users recently experienced issues searching for volunteer opportunities. Our search feature uses location data to find nearby opportunities. A recent update caused the system to make a large number of requests to our geoocation service, which unfortunately led to a temporary overload. This meant some users encountered an error message while searching, while others were able to search successfully. ### Root Cause A temporary issue with our geolocation service caused some users to experience difficulties. This was due to an unexpectedly large number of requests, which overloaded the geolocation service. ### Future Mitigation To prevent this issue from recurring, we're making some improvements to how we handle location information. Going forward, the system will only store a user's full address when they specifically update it. This allows us to still display opportunities that are closest to the user.