Jenzabar Outage History

Jenzabar Cloud Services detected a host failure in our North America Central data center location. The host has been taken out of service at this time, and all customer workloads carried by this host have been restarted on other hosts within the cluster.

Cloud Infrastructure engineers have been closely monitoring a router's health in our North America East IBM data center region. Currently there are no known outages, but due to growing concerns for this particular router, we will be performing a failover to a secondary unit so that further troubleshooting and corrective actions can be taken before the situation worsens. Failover Scheduled When: September 22nd @ 11:45am EST Duration: 0-20 seconds What to expect: at most, a very brief interruption of public traffic could be expected. Internal systems will remain connected to one another during this process. This issue does not affect our other Data Center locations. We apologize for any inconvenience. Cloud Infrastructure

Jenzabar's Cloud Infrastructure team has identified a memory issue on 2 of the edge nodes within the firewall/VPN cluster in our Dallas location. In order to correct this issue, services will be transitioned to other edge nodes within the cluster at 12:00pm EST. This will cause a brief VPN outage (1-5 seconds) for customers currently running on either of these two edge nodes. We apologize for the inconvenience.

Jenzabar Cloud Infrastructure has identified and isolated improperly functioning hardware and is currently working with IBM support to take corrective action. Additionally, an unexpected failover event took place at the following time: 2024/12/10 15:11:37 EST Cloud infrastructure isolated the bad hardware and performed another failover at: 2024/12/10 15:47:12 Disconnects to hosted resources could have occurred during the times mentioned above. At this time, we expect all services to remain stable. Please stay tuned as we will continue to provide updates as they are available.

Emergency network configuration changes December 9th, starting at 9pm EST. We have discovered an issue with one of the edge firewalls in our North America East location that will require emergency work later tonight, December 9th, starting at around 9pm EST. Work should be concluded by 11pm EST. Currently, no outages have been caused by this issue. A re-route of network services will be triggered at some point during the maintenance window, and a router will be rebooted to commit those changes. While there is no anticipated impact on customer workloads during the maintenance, be advised that this work is being done during this time. An update will be sent out later tonight once the work has been completed.

At approximately 10:20am EST this morning, one of the VMware hosts failed due to a memory module/DIMM failure. All virtual servers running on that host would have been immediately automatically powered off, then powered back on once transferred to a different host within the cluster. Total downtime would have been approximately 1-3 minutes. All systems should now be functioning normally, and the failed host will be repaired and brought back into production later today.

IBM engineers are engaged to repair the failed memory module on the offline host. All systems continue to function normally at this time.

Although IBM also listed outages at the same time, we have determined a CrowdStrike update is the cause. Many of our clients are ok and online and did not experience any issues because the update did not make it to all systems. Crowdstrike has provided us with steps to resolve the problem. Our team is working to manually apply the fix to each system.

The team is making good progress applying the fix Crowdstrike provided. We are almost finished with the North America East data center. Access is being restored as quickly as possible.

We are continuing to work on a fix for this issue.