CampBrain experienced a critical incident on March 3, 2025 affecting Office Portal and Parent Portal and 1 more component, lasting 3h 4m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 03, 2025, 05:57 PM UTC
We are currently experiencing some slowness across our portals. We are investigating the issue. Please avoid taking actions as they may result in errors. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.
- investigating Mar 03, 2025, 06:12 PM UTC
We continue to experience ongoing connectivity issues with the CampBrain portals due to a wide-spread Microsoft Azure outage. Our Developers have escalated the issue with Microsoft Azure engineers to gather more information and identify a timeline. In the meantime, we will be taking all portals offline (Office, Registration, Staff, Giving and Booking Portals). We sincerely apologize for the inconvenience and we'll continue to update you as more information becomes available.
- identified Mar 03, 2025, 08:06 PM UTC
Microsoft Azure have identified and mitigated a network issue that was causing their wide-spread outage. We are monitoring to ensure that we see steady improvement to all of our portals. We are in the process of bringing the Office Portal back online and will be monitoring for a short period before bringing the other portals (Registration, Staff, Giving, Booking) back online. Again, we apologize and thank you for your patience during this disruption in service.
- monitoring Mar 03, 2025, 08:21 PM UTC
We are seeing improvements on all CampBrain portals, and have brought the Parent, Staff, Giving, and Booking portals back online. We will be monitoring all portals closely to ensure that things are returning to normal. Thank you for your patience as we have worked through this Microsoft Azure service issue.
- resolved Mar 03, 2025, 09:01 PM UTC
This incident has been resolved.
- postmortem Mar 06, 2025, 05:03 PM UTC
On March 3rd starting shortly after 12:30 PM EST, CampBrain Engineers were alerted of longer than normal response times across all CampBrain portals. Upon investigation, we identified that the response time was a result of a wide-spread Microsoft Azure outage impacting networking services our infrastructure relies upon. In the interim, changes to data were taking a long time to propagate and we made the decision to take our portals offline \(the Office Portal, Registration Portal, Giving Portal, Staff Portal and Booking Portal\) as to stop the poor user experience on any of these portals. At 3:00 PM EST, Microsoft Azure had applied a fix within their services, and we observed steady improvements across our services. We were able to bring all of our portals back online by 3:20 PM EST. We understand the impact these recent incidents have had on you. As summer approaches, we know that system reliability is the highest priority. The CampBrain team's priority is to make sure that you can confidently trust CampBrain. We sincerely and profoundly apologize and we’re grateful for your understanding and patience.