Bloomerang experienced a critical incident on January 9, 2025 affecting CRM App - crm.bloomerang.co and CRM API - api.bloomerang.co, lasting 2h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 09, 2025, 03:02 AM UTC
We have been experiencing a Bloomerang service outage since around 9:15pm EST. Customers may see errors when attempting to login and interact with the application. Our team is currently working to restore fucntionality. We apologize for any inconvenience. An update will be provided in 15 minutes.
- identified Jan 09, 2025, 03:26 AM UTC
We are actively working to restore functionality for our users. All components have been affected including online giving forms. Embedded, Hosted, and Self-Hosted online forms are unable to load during this outage and may display a "Donation Form could not load. Please try again later." An update will be provided in 15 minutes.
- identified Jan 09, 2025, 03:44 AM UTC
As of 10:38pm EST we have identified root cause and functionality has been restored. Online forms are loading as expected now. Users will now be able to login but may see degraded performance in the application until further notice. An update will be provided in 15 mins.
- monitoring Jan 09, 2025, 04:07 AM UTC
A fix has been implemented and we are monitoring results.
- monitoring Jan 09, 2025, 04:27 AM UTC
We are continuing to monitor the application for any further issues.
- resolved Jan 09, 2025, 05:11 AM UTC
This incident has been resolved.
- postmortem Jan 15, 2025, 09:59 PM UTC
## **System Outage Postmortem** **Date and Time of Incident:** January 9, 2025, 9:15 PM to 10:35 PM Eastern **Incident Summary:** On January 9, our system experienced an outage from 9:15 PM to 10:35 PM Eastern, impacting several critical services. During this time, some users may have been unable to log into the application and/or experienced errors throughout the platform. Donors attempting to complete transactions via donation forms encountered errors, and API services were disrupted. The outage was caused by an improper configuration value for our CRM services. This misconfiguration prevented CRM services from starting correctly, resulting in disruptions across the system. To resolve the issue, we corrected the named configuration value, which restored service performance. **Next Steps:** To prevent similar incidents, we are improving alignment on configuration changes across teams to ensure thorough review and verification prior to implementation. We sincerely apologize for this disruption and appreciate your patience as we work to strengthen our systems and processes.