Slowness & Errors

Update timeline

1. investigating Feb 27, 2025, 03:44 PM UTC

   We are currently experiencing some slowness across our portals. We are investigating the issue. Please avoid taking actions as they may result in errors. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. investigating Feb 27, 2025, 04:26 PM UTC

   We continue to experience slowness across Portals. We have made the decision to take the Parent, Staff, and Giving portals offline again. CampBrain Office Portal will remain online and should be considered to be in read-only mode. Please continue to avoid taking actions as they may result in errors. Thank you for your patience and understanding.

3. monitoring Feb 27, 2025, 05:38 PM UTC

   Following a thorough investigation by our engineering team, we have made the decision to roll back our recent release due to concerns about infrastructure stability. While any data you have entered remains unaffected, certain new features—such as the recently introduced mail merge fields—will no longer be available at this time. The Parent, Staff, and Giving portals are now back online. We will be monitoring all portals closely to ensure that things are returning to normal. We will provide further details on these events and our plans for an updated release timeframe to you in the coming days. Thank you for your patience and understanding as we work to ensure a stable and reliable experience.

4. resolved Feb 27, 2025, 06:33 PM UTC

   This incident has been resolved.

5. postmortem Mar 03, 2025, 04:38 PM UTC

   To CampBrain clients: On February 26th starting around 10:00 AM EST, CampBrain Engineers were alerted of longer than normal response times across all CampBrain portals. We quickly identified which services were causing the performance degradation and began restarting those services, as this is a common and reliable fix to this issue. After restarting those services, we identified that this fix did not yield positive results and continued to experience long response times and errors. Our team engaged our hosting platform, Microsoft Azure, for assistance. In the interim, changes to data were taking a long time to propagate and we made the decision to take our public facing portals offline \(the Registration Portal, Giving Portal, Staff Portal and Booking Portal\) as to reduce the poor user experience on any of these portals. Shortly after 1:00 PM EST, our engineering saw performance levels return to normal and were able to bring all portals back online. Our engineers along with our contact at Microsoft continued to investigate as they recognized that the root cause had not yet been addressed and remained committed to continuing their diagnosis. On February 27th starting around 10:30 AM EST, CampBrain Engineers once again noted a degradation in response time across all portals. We continued our investigation and dialogue with Microsoft, and took the Registration Portal, Staff Portal, Giving Portal and Booking Portal offline. After further investigation, we made the difficult decision to roll back our recent release, which launched on February 26th. Further investigations around the root causes pointed to infrastructure changes made with 2025.02 which confirmed that the rollback was the right decision to make. At approximately 12:30 PM EST, we reverted back to version 2024.12 of CampBrain which saw immediate relief of the issues and we were able to put all portals back online. With this in mind, we will not re-release immediately but instead, will be including the changes and features from this release into our next release, which is currently planned for the end of April. This allows our team to fully engage in a post-mortem process which will identify strategies for re-releasing the infrastructure changes without impacting the performance of the system. We are very sorry for the disruptions over February 26th and 27th. We understand how much you, your organization, and your families rely on CampBrain. It is an incredibly difficult decision to roll back a release. Like many of you, our team is equally excited about new features and introducing behind-the-scenes infrastructure improvements. We have worked hard to get these items out to you so you can get the most out of CampBrain. We feel as much responsibility to you as you do to your community. Thank you for your patience as we work to discover the root cause, learn more, and work to improve. We understand that ensuring the stability of CampBrain is paramount – both for yourselves and your registrants. Thank you, CampBrain Leadership \(Alison, Alison, Jeff & Mayuran\)