CampBrain experienced a minor incident on July 9, 2024 affecting Office Portal, lasting 36m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 09, 2024, 12:53 PM UTC
We are currently experiencing an issue with reports failing to generate. Our Development team is investigating and will post updates here as soon as they become available.
- monitoring Jul 09, 2024, 01:16 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 09, 2024, 01:30 PM UTC
This incident has been resolved.
- postmortem Jul 12, 2024, 07:03 PM UTC
**Summary** On Tuesday, July 9th, our reporting servers failed to come online resulting in CampBrain Reporting being unavailable from 6 am – 9:15 am eastern. This resulted in the inability to access standard and custom reports, accessing any system reports and sending e-mails with reports attached. Our automatic alerting system for critical issues did not alert our DevOps team to the issue so our response time to resolving the issue was delayed. **Moving Forward** We have engaged with the vendor of our reporting tool to understand why this occurred. Additionally, we are developing automatic alerts to provide early warning to our development team should this scenario occur again. By having alerts available, we can respond quicker and reduce the time of the outage. We sincerely and profoundly apologize for the multi-hour disruption on Tuesday during a critical time of your operations. We know that system reliability is the highest priority so we will continue to ensure our infrastructure is solid so we can reduce the risk to you and your participants. We’re grateful for your understanding and patience.