CampBrain Outage History

Started Nov 05, 2026, 09:02 PM UTC· resolved Nov 05, 2026, 10:35 PM UTC

1. investigating Nov 05, 2026, 09:02 PM UTC

   We are getting reports that payments made with credit cards through the Shift4/Worldline payment processor are failing. We contacted Shift4/Worldline who is investigating. We will provide updates here as we learn more.

2. monitoring Nov 05, 2026, 10:18 PM UTC

   Shift4/Worldline have applied a fix which has resolved the credit card processing issues impacting some clients. We're continuing to monitor and will provide an update if anything changes.

3. resolved Nov 05, 2026, 10:35 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 17, 2026, 02:21 PM UTC· resolved Sep 17, 2026, 03:37 PM UTC

1. investigating Sep 17, 2026, 02:21 PM UTC

   We are currently experiencing an issue with reports failing to generate. Our Development team is investigating and will post updates here as soon as they become available.

2. monitoring Sep 17, 2026, 02:46 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 17, 2026, 03:37 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 20, 2026, 02:33 PM UTC· resolved Jul 20, 2026, 05:54 PM UTC

1. investigating Jul 20, 2026, 02:33 PM UTC

   Some clients are experiencing intermittent slowness in the Office Portal. We are investigating the issue and will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. investigating Jul 20, 2026, 03:16 PM UTC

   Our team continues to investigate the root cause of the slowness affecting the Office Portal. We will continue to monitor the situation closely and provide further updates as our investigation progresses.

3. investigating Jul 20, 2026, 04:24 PM UTC

   Our engineering team has identified a degradation in Microsoft Azure services which is causing slowness across CampBrain portals. We are working to understand a timeline around service recovery from Microsoft Azure. We are also investigating the viability of bypassing affected services. Thank you for your patience.

4. identified Jul 20, 2026, 04:29 PM UTC

   Our engineering team has identified a degradation in Microsoft Azure services which is causing slowness across CampBrain portals. We are working to understand a timeline around service recovery from Microsoft Azure. We are also investigating the viability of bypassing affected services. Thank you for your patience.

5. monitoring Jul 20, 2026, 04:57 PM UTC

   Our engineering team has deployed a fix intended to work around the degraded Microsoft Azure services, and we have seen response times drop back to normal values in CampBrain portals. We are continuing to monitor performance closely and will update or resolve this status page as appropriate. Thank you for your patience as we have been working on this problem.

6. resolved Jul 20, 2026, 05:54 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 11, 2026, 05:13 PM UTC· resolved Mar 11, 2026, 05:39 PM UTC

1. investigating Mar 11, 2026, 05:13 PM UTC

   We are currently experiencing slowness running reports and are investigating the issue.

2. monitoring Mar 11, 2026, 05:21 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 11, 2026, 05:39 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 10, 2026, 04:58 PM UTC· resolved Mar 10, 2026, 05:28 PM UTC

1. investigating Mar 10, 2026, 04:58 PM UTC

   We are currently experiencing an increase in system errors across our portals. We are investigating the issue and will provide more information soon.

2. identified Mar 10, 2026, 05:05 PM UTC

   We have identified the issue and implemented mitigation measures. Occurrences of system errors are declining as we continue to monitor.

3. monitoring Mar 10, 2026, 05:21 PM UTC

   A fix has been implemented and we are continuing to monitor the results.

4. resolved Mar 10, 2026, 05:28 PM UTC

   This incident has been resolved.

5. postmortem Mar 10, 2026, 06:11 PM UTC

   At 12:15 PM EST, an error with one of our background processes led to a rise in system errors across our portals. Our automated alert system brought this to our attention. At 12:50 PM EST, we restarted our receivers as a mitigation measure, which reduced the volume of system errors immediately. By 12:58 PM EST, a specific background process was identified as the root cause, and was fixed. Moving forward, continued improvements to our infrastructure and internal tooling will help us minimize occurrences of these types of errors. In the meantime, we will swiftly identify and resolve these errors in a similar fashion to today’s incident.

Read the full incident report →

Started Jan 13, 2026, 03:40 PM UTC· resolved Jan 14, 2026, 07:59 PM UTC

1. investigating Jan 13, 2026, 03:40 PM UTC

   We are currently experiencing some slowness in the office and parent portals. We are investigating the issue. The timeline for a fix is unknown at this point, but we will provide updates as soon as possible.

2. investigating Jan 13, 2026, 04:22 PM UTC

   We have implemented various mitigation measures and seen response times improve. We continue to investigate the root cause of the issue, and the timeline for a fix remains unknown.

3. investigating Jan 13, 2026, 06:00 PM UTC

   Response times across the parent and office portals continue to show signs of improvement. Further short-term mitigation measures are being explored while the investigation into the root cause persists. We will provide another update at 2PM EST.

4. investigating Jan 13, 2026, 07:04 PM UTC

   Response times across the parent and office portals have returned to normal, and we are closely monitoring for any changes to that status. Further short-term mitigation measures are being explored while the investigation into the root cause persists. We will provide another update at 4PM EST.

5. investigating Jan 13, 2026, 08:55 PM UTC

   Parent and Office Portal performance continues to hold steady, and we are not seeing the elevated response times reported earlier today. While we continue investigating the underlying root cause, we are moving forward with additional stability improvements this evening to further improve responsiveness and ensure a smoother end-user experience. We will share another update at 6:00 PM EST.

6. investigating Jan 13, 2026, 10:43 PM UTC

   Parent and Office Portal performance continues to hold steady. We will be patching overnight tonight to address stability and improve the registration experience. We will share another update by January 14th, 2026 at 8:30 AM ET.

7. monitoring Jan 14, 2026, 01:19 PM UTC

   After a successful patch and implementing various mitigation measures, response times have returned to normal. We are monitoring the situation closely and will share another update by 11:00 AM EST.

8. monitoring Jan 14, 2026, 03:55 PM UTC

   We have been monitoring this morning's traffic and response times have remained normal, further indicating a successful patch and mitigation efforts. We will continue live monitoring and provide another update at 1:00 PM EST.

9. monitoring Jan 14, 2026, 05:59 PM UTC

   Traffic and response times have remained normal. We will continue live monitoring and provide another update at 3:00 PM EST.

10. resolved Jan 14, 2026, 07:59 PM UTC

    Performance has stabilized and the system is operating normally. We will continue to monitor closely and will share a post-mortem by 5:00 pm EST.

11. postmortem Jan 14, 2026, 09:41 PM UTC

    Since early January, we experienced periods of slowness impacting the Registration and Office Portals, resulting in elevated response times and major registration disruption for some users. Specifically: * Slow response times and loading on the Registration Portal. * Errors as registrants navigate the Registration Portal. * Being unexpectedly logged out or “kicked out” during registration. * Slowness within the Office Portal. Our findings indicate these disruptions were driven by periods of elevated traffic in the Registration Portal that exceed optimal operating levels. We completed a patch on January 13th, focused on some of the areas causing the most friction during the registration process. We closely monitored and adjusted waiting rooms to help manage traffic during high-volume periods and ensure system stability in the Registration Portal.

Read the full incident report →

Started Jan 12, 2026, 02:36 PM UTC· resolved Jan 12, 2026, 04:24 PM UTC

1. investigating Jan 12, 2026, 02:36 PM UTC

   We are currently experiencing some slowness in the parent portal. We are investigating the issue. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. monitoring Jan 12, 2026, 02:57 PM UTC

   The issue has been identified and a fix is being implemented. We are seeing improved response times in the parent portal and will continue to monitor the situation closely.

3. monitoring Jan 12, 2026, 03:50 PM UTC

   We continue to see improved response times and are monitoring the situation closely.

4. resolved Jan 12, 2026, 04:24 PM UTC

   After implementing various mitigation measures, response times have returned to normal. We are confident this incident has been resolved.

Read the full incident report →

Started Jan 05, 2026, 04:38 PM UTC· resolved Jan 06, 2026, 02:08 PM UTC

1. investigating Jan 05, 2026, 04:38 PM UTC

   We are currently experiencing slowness in the Office and Parent Portals. We are investigating and will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. identified Jan 05, 2026, 04:51 PM UTC

   We are seeing improved response times across both the Office and Parent Portals. We continue to monitor closely.

3. identified Jan 05, 2026, 05:40 PM UTC

   We are seeing improvements within the Office Portal but do still experience slowness when saving changes to the database (ie. saving a manual registration). We also are seeing that the Parent Portal is still experiencing slowness with navigation and submitting applications. Our team continues to investigate and are actively applying fixes.

4. identified Jan 05, 2026, 06:14 PM UTC

   We continue to see performance improvements within the Office Portal including saving changes. We continue to actively work on improving the performance on the Parent Portal which continues to experience slower performance with navigation and submitting applications.

5. monitoring Jan 05, 2026, 10:26 PM UTC

   Our Engineering team continues to work on and monitor performance issues impacting the Parent Portal. We have seen improvement in performance and have plans in place for an overall solution. We’ve identified multiple contributing factors, including a bug affecting Custom Forms. This bug is causing delays when saving form data, which in turn increases system load and results in slower performance during peak usage times. Our team is actively working to resolve this issue. If you plan to go live before the fix is in place, you can reduce the impact by either temporarily making Custom Forms unavailable during registration (allowing registrants to complete the Form later) or by temporarily unlinking configurable fields from custom forms. This is a top priority to solve and we will provide an update when this bug has been fixed. In parallel, we are monitoring system performance closely and making enhancements to the tools and infrastructure that support the application. Additional related items, including slower application submissions that may appear in a “processing” state longer than expected, are also under review. We will continue to share more information as we work on fixes. Thank you for your patience and understanding.

6. resolved Jan 06, 2026, 02:08 PM UTC

   We have continued to see performance improvements on the Registration Portal. We had identified multiple contributing factors, including a bug affecting Custom Forms. This bug is causing delays when saving form data, which in turn increases system load and results in slower performance during peak usage times. Our team is actively working to resolve this bug. If you plan to go live before the fix is in place, you can reduce the impact by either temporarily making Custom Forms unavailable during registration (allowing registrants to complete the Form later) or by temporarily unlinking configurable fields from custom forms. This is a top priority to solve and we will provide an update when this bug has been fixed. We will keep you up-to-date on the status of the bug via a notice within CampBrain.

7. postmortem Jan 09, 2026, 08:39 PM UTC

   **An Update on System Reliability** Many of us returned to work this week after some well-deserved time off, ready to hit the ground running in 2026. For many of you, this meant launching registration and completing time-sensitive work in CampBrain. Instead, CampBrain experienced a system-wide disruption that impacted both the Registration and Office Portals. This week was a critical, high-traffic period for our users, and our software did not perform when you needed it most. This caused significant unexpected stress for you, your registrants, and our team, and we know this is on us. We also recognize that this was not an isolated incident. When these unexpected issues arise repeatedly during peak periods, it erodes trust. And we have significantly impacted the trust you place in us. The recent incident was specifically triggered by a bug within Person Custom Forms, but it highlighted areas of the platform where we need to accelerate our investment in improving performance and reliability. We are beginning that work immediately to ensure the system consistently performs during your busiest and most critical moments. **What’s Changing** We are making a deliberate shift in priorities, effective immediately: · **We are doubling down our software development team’s efforts on system reliability and performance.** We’re putting additional resources towards strengthening the foundation of the platform. A small team of developers will continue working on feature additions and improvements, but system performance is our #1 priority. · **We are addressing performance bottlenecks.** This work is specifically aimed at improving stability during the busiest and most critical moments of your season. · **We are setting clear performance and reliability targets.** We will not stop working on performance until we consistently and confidently meet these targets during real-world peak usage. · **This effort is being monitored and led by the CampBrain Team and beyond.** Our CampBrain Leadership and leadership across our parent organization are actively involved. **What You Can Expect** · An immediate, significant effort to reduce disruptions during critical registration and operational periods · A prioritized effort to ensure faster, more predictable system performance · Clear, proactive updates on progress, with defined goals and measurable benchmarks We recognize that apologies are not enough. What matters is visible, meaningful improvement in CampBrain as a result of our immediate actions. We are committed to being transparent with you and sharing updates as this work progresses. We want to be clear about what is changing and why. CampBrain exists to support the important work you do. It should make your job easier, so you have more time to focus on what matters – delivering memorable experiences to your program participants. Restoring reliability and your confidence in us is essential, and it is the work we are focused on right now. Thank you for your continued partnership. We promise to make sustained improvements to rebuild your trust. Thank you, Jon Winer and Mayuran Sunderam CampBrain Engineering Leaders

Read the full incident report →

Started Dec 03, 2025, 01:11 PM UTC· resolved Dec 03, 2025, 03:36 PM UTC

1. investigating Dec 03, 2025, 01:11 PM UTC

   We are currently experiencing an issue with processing credit card and eCheck payments via CampBrain Payments. Our Engineering Team is investigating.

2. identified Dec 03, 2025, 01:21 PM UTC

   Our engineers have identified the source of the issue and are working on a fix. Please note, this is only impacting our in-house payment processor. Those processing via Worldline/Bambora for credit cards or Forte for eChecks are not impacted.

3. monitoring Dec 03, 2025, 03:07 PM UTC

   Our team has implemented a fix and we are seeing CampBrain Payments transactions processing as normal again. We are continuing to monitor the situation and will update or resolve this status in the next 30 minutes.

4. resolved Dec 03, 2025, 03:36 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 18, 2025, 02:24 PM UTC· resolved Nov 18, 2025, 03:14 PM UTC

1. investigating Nov 18, 2025, 02:24 PM UTC

   CampBrain Payments users may be unable to load payment fields in any CampBrain portal due to an ongoing issue with Cloudflare, who host key components of the CampBrain Payments infrastructure. Their status page can be viewed here: https://www.cloudflarestatus.com, and we will also keep this status updated as we learn more.

2. monitoring Nov 18, 2025, 02:47 PM UTC

   Cloudflare has implemented a fix and we are seeing CampBrain Payments fields loading normally again. Our team is continuing to monitor the situation and will update or resolve this status in the next 30 minutes.

3. resolved Nov 18, 2025, 03:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 11, 2025, 06:57 PM UTC· resolved Nov 11, 2025, 08:01 PM UTC

1. investigating Nov 11, 2025, 06:57 PM UTC

   We are currently experiencing slowness across all portals. We are investigating the issue. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. monitoring Nov 11, 2025, 07:39 PM UTC

   A fix has been implemented and we are seeing improved response times. We will continue to monitor closely.

3. resolved Nov 11, 2025, 08:01 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 29, 2025, 04:33 PM UTC· resolved Oct 29, 2025, 11:44 PM UTC

1. investigating Oct 29, 2025, 04:33 PM UTC

   We are currently experiencing some slowness across our portals as a result of an Microsoft Azure degradation of service that may impact some users. We are investigating the issue and we will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. identified Oct 29, 2025, 06:31 PM UTC

   Microsoft Azure has identified the root cause of the degradation of service and is in the process of correcting the issue. Slowness may continue for some users, but we are anticipating improved service in the next hour. We will provide another update as soon as more information becomes available. Thank you for your patience and understanding.

3. identified Oct 29, 2025, 07:32 PM UTC

   Microsoft Azure has provided a new update with firmer timelines around full service recovery. Service to our portals will begin to improve, but unfortunately, Microsoft's new report states that a full service recovery is expected within the next 4 hours. We will continue to monitor and provide new updates as necessary as they become available. Thank you again for your patience.

4. monitoring Oct 29, 2025, 08:30 PM UTC

   We are noticing widespread improvement in service performance. Some users may continue to experience service degradation, as Microsoft Azure has yet to provide an update on their full resolution ETA of 7:20pm ET. We are continuing to monitor and will publish another update when more information becomes available. Thank you again for your patience today.

5. resolved Oct 29, 2025, 11:44 PM UTC

   Microsoft is reporting 98% availability and that a majority of customers are seeing improvements across affected regions. Our metrics and tests are indicating that performance is stable.

Read the full incident report →

Started Jul 08, 2025, 03:49 PM UTC· resolved Jul 08, 2025, 05:00 PM UTC

1. investigating Jul 08, 2025, 03:49 PM UTC

   We are currently experiencing some slowness in the office portal. We are investigating the issue. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. investigating Jul 08, 2025, 04:19 PM UTC

   Our team continues to investigate the root cause of the slowness affecting the Office Portal. In the meantime, we’ve implemented mitigation measures that are resulting in improved response times. We will continue monitoring the situation closely and provide further updates as our investigation progresses.

3. monitoring Jul 08, 2025, 04:40 PM UTC

   We continue to see improved response times and are monitoring the situation closely. We will provide further updates as soon as possible.

4. resolved Jul 08, 2025, 05:00 PM UTC

   After implementing various mitigation measures, response times have returned to normal. We are confident this incident has been resolved.

Read the full incident report →

Started Jul 07, 2025, 03:10 PM UTC· resolved Jul 07, 2025, 03:56 PM UTC

1. investigating Jul 07, 2025, 03:10 PM UTC

   We are currently experiencing some slowness in the office portal. We are investigating the issue. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. identified Jul 07, 2025, 03:24 PM UTC

   We have identified the root cause of slowness in the office portal and are actively working on a fix.

3. monitoring Jul 07, 2025, 03:43 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 07, 2025, 03:56 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 17, 2025, 01:13 PM UTC· resolved Jun 17, 2025, 11:07 PM UTC

1. investigating Jun 17, 2025, 01:13 PM UTC

   We are currently experiencing some slowness across our portals. We are investigating the issue. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. investigating Jun 17, 2025, 01:58 PM UTC

   We are continuing to experience slowness within the Office Portal and the Camp Store. Our team is investigating and we will provide further updates as soon as more information becomes available. Thank you for your patience and understanding.

3. investigating Jun 17, 2025, 02:42 PM UTC

   Our team continues to investigate the root cause of the slowness affecting the Office Portal and Camp Store. In the meantime, we’ve implemented mitigation measures that are resulting in improved response times. We will continue monitoring the situation closely and provide further updates as our investigation progresses.

4. monitoring Jun 17, 2025, 03:52 PM UTC

   We continue to see improved response times and are monitoring the situation closely. We will provide further updates as our investigation progresses.

5. monitoring Jun 17, 2025, 07:00 PM UTC

   After implementing mitigation measures we are seeing improved, but not yet optimal response times on the Office Portal and Camp Store. While these steps are helping stabilize performance in the short term, our team is continuing to investigate the root cause to ensure a long-term resolution to the underlying performance issues. Please note there was a brief period of slowness earlier this afternoon due to a midday patch. This is now complete and we are once again seeing response times improving. We appreciate your patience as we work toward a permanent solution.

6. monitoring Jun 17, 2025, 10:14 PM UTC

   The system remains stable following our earlier mitigation efforts, with improved response times in the Office Portal. Users can continue to use CampBrain as normal. Our engineering team continues to monitor closely while investigating longer term improvements.

7. resolved Jun 17, 2025, 11:07 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 03, 2025, 05:57 PM UTC· resolved Mar 03, 2025, 09:01 PM UTC

1. investigating Mar 03, 2025, 05:57 PM UTC

   We are currently experiencing some slowness across our portals. We are investigating the issue. Please avoid taking actions as they may result in errors. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. investigating Mar 03, 2025, 06:12 PM UTC

   We continue to experience ongoing connectivity issues with the CampBrain portals due to a wide-spread Microsoft Azure outage. Our Developers have escalated the issue with Microsoft Azure engineers to gather more information and identify a timeline. In the meantime, we will be taking all portals offline (Office, Registration, Staff, Giving and Booking Portals). We sincerely apologize for the inconvenience and we'll continue to update you as more information becomes available.

3. identified Mar 03, 2025, 08:06 PM UTC

   Microsoft Azure have identified and mitigated a network issue that was causing their wide-spread outage. We are monitoring to ensure that we see steady improvement to all of our portals. We are in the process of bringing the Office Portal back online and will be monitoring for a short period before bringing the other portals (Registration, Staff, Giving, Booking) back online. Again, we apologize and thank you for your patience during this disruption in service.

4. monitoring Mar 03, 2025, 08:21 PM UTC

   We are seeing improvements on all CampBrain portals, and have brought the Parent, Staff, Giving, and Booking portals back online. We will be monitoring all portals closely to ensure that things are returning to normal. Thank you for your patience as we have worked through this Microsoft Azure service issue.

5. resolved Mar 03, 2025, 09:01 PM UTC

   This incident has been resolved.

6. postmortem Mar 06, 2025, 05:03 PM UTC

   On March 3rd starting shortly after 12:30 PM EST, CampBrain Engineers were alerted of longer than normal response times across all CampBrain portals. Upon investigation, we identified that the response time was a result of a wide-spread Microsoft Azure outage impacting networking services our infrastructure relies upon. In the interim, changes to data were taking a long time to propagate and we made the decision to take our portals offline \(the Office Portal, Registration Portal, Giving Portal, Staff Portal and Booking Portal\) as to stop the poor user experience on any of these portals. At 3:00 PM EST, Microsoft Azure had applied a fix within their services, and we observed steady improvements across our services. We were able to bring all of our portals back online by 3:20 PM EST. We understand the impact these recent incidents have had on you. As summer approaches, we know that system reliability is the highest priority. The CampBrain team's priority is to make sure that you can confidently trust CampBrain. We sincerely and profoundly apologize and we’re grateful for your understanding and patience.

Read the full incident report →

Started Mar 03, 2025, 05:01 PM UTC· resolved Mar 03, 2025, 07:56 PM UTC

1. investigating Mar 03, 2025, 05:01 PM UTC

   We are currently experiencing an issue with our phone system. We will be unable to take support phone calls until it is resolved. In the meantime, please contact [email protected] for support issues or questions. We're sorry for the inconvenience, and hope to have the issue resolved as soon as possible.

2. monitoring Mar 03, 2025, 07:01 PM UTC

   We were able to move our phone provider off of the Microsoft Azure service which means that our phone lines are once again working and you can call into our Support line as needed.

3. resolved Mar 03, 2025, 07:56 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 27, 2025, 03:44 PM UTC· resolved Feb 27, 2025, 06:33 PM UTC

1. investigating Feb 27, 2025, 03:44 PM UTC

   We are currently experiencing some slowness across our portals. We are investigating the issue. Please avoid taking actions as they may result in errors. We will provide updates as soon as more information becomes available. Thank you for your patience and understanding.

2. investigating Feb 27, 2025, 04:26 PM UTC

   We continue to experience slowness across Portals. We have made the decision to take the Parent, Staff, and Giving portals offline again. CampBrain Office Portal will remain online and should be considered to be in read-only mode. Please continue to avoid taking actions as they may result in errors. Thank you for your patience and understanding.

3. monitoring Feb 27, 2025, 05:38 PM UTC

   Following a thorough investigation by our engineering team, we have made the decision to roll back our recent release due to concerns about infrastructure stability. While any data you have entered remains unaffected, certain new features—such as the recently introduced mail merge fields—will no longer be available at this time. The Parent, Staff, and Giving portals are now back online. We will be monitoring all portals closely to ensure that things are returning to normal. We will provide further details on these events and our plans for an updated release timeframe to you in the coming days. Thank you for your patience and understanding as we work to ensure a stable and reliable experience.

4. resolved Feb 27, 2025, 06:33 PM UTC

   This incident has been resolved.

5. postmortem Mar 03, 2025, 04:38 PM UTC

   To CampBrain clients: On February 26th starting around 10:00 AM EST, CampBrain Engineers were alerted of longer than normal response times across all CampBrain portals. We quickly identified which services were causing the performance degradation and began restarting those services, as this is a common and reliable fix to this issue. After restarting those services, we identified that this fix did not yield positive results and continued to experience long response times and errors. Our team engaged our hosting platform, Microsoft Azure, for assistance. In the interim, changes to data were taking a long time to propagate and we made the decision to take our public facing portals offline \(the Registration Portal, Giving Portal, Staff Portal and Booking Portal\) as to reduce the poor user experience on any of these portals. Shortly after 1:00 PM EST, our engineering saw performance levels return to normal and were able to bring all portals back online. Our engineers along with our contact at Microsoft continued to investigate as they recognized that the root cause had not yet been addressed and remained committed to continuing their diagnosis. On February 27th starting around 10:30 AM EST, CampBrain Engineers once again noted a degradation in response time across all portals. We continued our investigation and dialogue with Microsoft, and took the Registration Portal, Staff Portal, Giving Portal and Booking Portal offline. After further investigation, we made the difficult decision to roll back our recent release, which launched on February 26th. Further investigations around the root causes pointed to infrastructure changes made with 2025.02 which confirmed that the rollback was the right decision to make. At approximately 12:30 PM EST, we reverted back to version 2024.12 of CampBrain which saw immediate relief of the issues and we were able to put all portals back online. With this in mind, we will not re-release immediately but instead, will be including the changes and features from this release into our next release, which is currently planned for the end of April. This allows our team to fully engage in a post-mortem process which will identify strategies for re-releasing the infrastructure changes without impacting the performance of the system. We are very sorry for the disruptions over February 26th and 27th. We understand how much you, your organization, and your families rely on CampBrain. It is an incredibly difficult decision to roll back a release. Like many of you, our team is equally excited about new features and introducing behind-the-scenes infrastructure improvements. We have worked hard to get these items out to you so you can get the most out of CampBrain. We feel as much responsibility to you as you do to your community. Thank you for your patience as we work to discover the root cause, learn more, and work to improve. We understand that ensuring the stability of CampBrain is paramount – both for yourselves and your registrants. Thank you, CampBrain Leadership \(Alison, Alison, Jeff & Mayuran\)

Read the full incident report →

Started Feb 26, 2025, 03:10 PM UTC· resolved Feb 26, 2025, 07:04 PM UTC

1. investigating Feb 26, 2025, 03:10 PM UTC

   We are currently experiencing some slowness in the office and registration portals. We are investigating the issue.

2. investigating Feb 26, 2025, 04:11 PM UTC

   We are receiving reports of error messages appearing after taking actions in the Office and Parent portals, as well as general slowness with the Office and Parent portals. We are continuing to investigate the problem.

3. investigating Feb 26, 2025, 04:47 PM UTC

   Our developers believe they have identified a problem between CampBrain and the hosting service, Microsoft Azure, and we are working with Azure engineers to solve the problem. We have made the decision to take the Parent, Staff, and Giving portals offline. CampBrain Office Portal will remain online and should be considered to be in read-only mode. Please avoid taking actions as they may result in errors. We will provide updates as soon as more information becomes available, thank you for your patience and understanding.

4. monitoring Feb 26, 2025, 06:28 PM UTC

   We are seeing improvements on all CampBrain portals, and are now bringing the the Parent, Staff, and Giving portals back online. We will be monitoring all portals closely to ensure that things are returning to normal. Thank you for your patience as we have worked through this service issue with Microsoft Azure.

5. resolved Feb 26, 2025, 07:04 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 14, 2025, 02:43 PM UTC· resolved Jan 14, 2025, 04:56 PM UTC

1. investigating Jan 14, 2025, 02:43 PM UTC

   We are currently experiencing slowness and error messages appearing in the CampBrain Office and Registration portals. We are investigating the issue.

2. monitoring Jan 14, 2025, 04:22 PM UTC

   A fix has been implemented and we are seeing response times within the Office and Parent Portals return to normal. Our team continues to monitor closely.

3. resolved Jan 14, 2025, 04:56 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 11, 2024, 03:53 PM UTC· resolved Dec 11, 2024, 06:19 PM UTC

1. identified Dec 11, 2024, 03:53 PM UTC

   We are currently experiencing an issue that will not allow users to push through in progress applications via the Office Portal. Our Team has identified the issue and are working on a resolution.

2. resolved Dec 11, 2024, 06:19 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 04, 2024, 05:39 PM UTC· resolved Dec 04, 2024, 06:59 PM UTC

1. investigating Dec 04, 2024, 05:39 PM UTC

   We are currently experiencing some slowness and error messages appearing in CampBrain registration portals. We are investigating the issue.

2. investigating Dec 04, 2024, 05:56 PM UTC

   We are currently experiencing some slowness and error messages appearing in CampBrain registration and office portals. We are investigating the issue.

3. monitoring Dec 04, 2024, 06:29 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 04, 2024, 06:59 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 22, 2024, 02:03 PM UTC· resolved Nov 22, 2024, 03:57 PM UTC

1. investigating Nov 22, 2024, 02:03 PM UTC

   We are currently experiencing some slowness in the office and registration portals. We are investigating the issue.

2. monitoring Nov 22, 2024, 03:30 PM UTC

   A fix has been implemented and we are monitoring.

3. resolved Nov 22, 2024, 03:57 PM UTC

   This incident has been resolved.

Read the full incident report →