Bloomerang experienced a critical incident on August 14, 2025 affecting CRM App - crm.bloomerang.co and Volunteer App - volunteer.bloomerang.co and 1 more component, lasting 19h 40m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 14, 2025, 09:40 PM UTC
We are currently experiencing an issue with the Bloomerang application and users are unable to login. Online forms and emails may be affected during this outage. We apologize for this inconvenience and will be providing an update in the next 30 mins.
- investigating Aug 14, 2025, 10:23 PM UTC
We are continuing to investigate this issue. Users will be unable to login until this is resolved. We will provide another update in the next 30 mins.
- investigating Aug 14, 2025, 11:01 PM UTC
We are continuing to investigate this issue. Users will be unable to login until this is resolved. We will provide another update as information becomes available.
- investigating Aug 14, 2025, 11:53 PM UTC
We are continuing to investigate this issue. Users will be unable to login until this is resolved. We will provide another update as information becomes available.
- investigating Aug 15, 2025, 12:17 AM UTC
Users are still unable to login to the Bloomerang application, as well as logging into Qgiv and Bloomerang Volunteer via Bloomerang. Online donation forms are also experiencing issues loading. We apologize for this inconvenience. Another update will be posted in 30 mins.
- identified Aug 15, 2025, 01:05 AM UTC
The issue has been identified and we are actively working to release a fix. In the meantime, users are still unable to login to the Bloomerang application, as well as logging into Qgiv and Bloomerang Volunteer via Bloomerang. Online donation forms are also experiencing issues loading. We apologize for this inconvenience and will be providing an update soon.
- identified Aug 15, 2025, 01:43 AM UTC
We are continuing to work on a fix for this issue.
- identified Aug 15, 2025, 02:48 AM UTC
We are continuing to work on a fix for this issue. In the meantime, users are still unable to login to the Bloomerang application, as well as logging into Qgiv and Bloomerang Volunteer via Bloomerang. Online donation forms are also experiencing issues loading. We will provide another update as information becomes available.
- identified Aug 15, 2025, 05:55 AM UTC
We are still continuing to work on a fix for this issue. At this time, users are still unable to login to the Bloomerang application, as well as logging into Qgiv and Bloomerang Volunteer via Bloomerang. Online donation forms are also experiencing issues loading. We will provide another update as information becomes available.
- identified Aug 15, 2025, 08:05 AM UTC
We are still continuing to work on a fix for this issue. At this time, users are still unable to login to the Bloomerang application, as well as logging into Qgiv and Bloomerang Volunteer via Bloomerang. Online donation forms are also experiencing issues loading. We appreciate your patience as we continue to work to restore service and will provide another update as information becomes available.
- monitoring Aug 15, 2025, 09:31 AM UTC
At this time, we have successfully implemented a fix for the issues affecting the Bloomerang application. All services are now fully restored, and we will continue to monitor the system to ensure stability. We sincerely apologize for any inconvenience this incident has caused and we appreciate your patience as we've worked to to restore service. If you continue to experience issues accessing Bloomerang, please contact us at [email protected]
- resolved Aug 15, 2025, 05:20 PM UTC
We have now confirmed that all services are stable and access to Bloomerang remains fully restored. This incident is now resolved. We appreciate your patience and understanding as we worked to resolve this incident and ensure system stability, and we apologize for any inconvenience. If you continue to experience issues accessing Bloomerang, please contact us at [email protected]
- postmortem Aug 25, 2025, 03:31 PM UTC
# Overview and Timeline On August 14, 2025, Bloomerang teams identified a critical software bug that would cause unexpected behavior in the CRM product. Upon discovery of this issue, our incident management team made the decision to temporarily disable customer access to the CRM application at 5:35 p.m. ET to prevent users from experiencing degraded service and application instability. We restored temporary access to enable critical integrations with Bloomerang Volunteer and Qgiv to continue functioning properly in the early stages of the incident. This allowed essential automated processes and third-party connections to operate while we continued working on the full resolution; although some donation forms were still impacted. Engineering teams identified the root cause and a fix was tested and deployed. A comprehensive fix was deployed to resolve the application behavior issue, and full service restoration was achieved at 5:19 a.m. ET on Friday morning. Teams continued monitoring to ensure complete resolution and application stability. # Root Cause\(s\) **A software bug within our CRM application code that caused unexpected behavior that required immediate preventative maintenance.** Upon discovery of these issues, teams made the decision to proactively take the system offline rather than risk widespread user experience degradation. # Action Item\(s\) | **Action Item** | **Tentative Completion Date** | | --- | --- | | Audit our development lifecycle to strengthen our release processes. | Friday, 9/26/2025 | | Develop improved rollback procedures for application code changes to reduce service restoration time in future incidents. | Friday, 9/26/2025 | | Document and test emergency maintenance procedures for rapid service isolation when critical application bugs are detected; as an example, this would allow us to keep donation forms operational while blocking user access to the application. | Multiple iterations are needed to solve this item overall, first iteration due by Friday, 9/26/2025 |