Jenzabar experienced a minor incident on July 19, 2024 affecting North America East and North America Central, lasting 1d 8h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 19, 2024, 08:28 AM UTC
IBM has reported problems at both Datacenters Jenzabar utilizes for our SaaS services. This has caused intermittent outages for all Jenzabar customers. As soon as we have more information from IBM, we will provide an update. This outage started around 2:00 AM EST
- identified Jul 19, 2024, 12:25 PM UTC
Although IBM also listed outages at the same time, we have determined a CrowdStrike update is the cause. Many of our clients are ok and online and did not experience any issues because the update did not make it to all systems. Crowdstrike has provided us with steps to resolve the problem. Our team is working to manually apply the fix to each system.
- identified Jul 19, 2024, 03:37 PM UTC
The team is making good progress applying the fix Crowdstrike provided. We are almost finished with the North America East data center. Access is being restored as quickly as possible.
- identified Jul 19, 2024, 04:44 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Jul 19, 2024, 04:45 PM UTC
We have a few production systems remaining and most client test systems. If you are still experiencing issues with a production system, please submit a ticket via https://support.jenzabar.net. Our team applied the changes needed to over 800 servers in five hours. Each system needed to be restarted into safe mode with networking support. A specific file was removed and then the system was restarted. We apologize for the inconvenience this has caused and we will continue to work to resolve any remaining issues.
- monitoring Jul 19, 2024, 07:31 PM UTC
We are continuing to monitor for any further issues.
- resolved Jul 20, 2024, 05:15 PM UTC
This incident has been resolved.