Jenzabar experienced a minor incident on December 10, 2024 affecting North America East, lasting 1d 17h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 10, 2024, 08:37 PM UTC
Addition issues were discovered with one of the edge firewalls in our North America East location. This a continuation of the issues discovered yesterday, December 9th, and we will be working to restore full functionality. At this time, all systems are up and operational, however there may be brief moments of dropped packets in flight between this cluster and the internet. We hope to have an update within the hour.
- identified Dec 10, 2024, 08:59 PM UTC
Jenzabar Cloud Infrastructure has identified and isolated improperly functioning hardware and is currently working with IBM support to take corrective action. Additionally, an unexpected failover event took place at the following time: 2024/12/10 15:11:37 EST Cloud infrastructure isolated the bad hardware and performed another failover at: 2024/12/10 15:47:12 Disconnects to hosted resources could have occurred during the times mentioned above. At this time, we expect all services to remain stable. Please stay tuned as we will continue to provide updates as they are available.
- monitoring Dec 11, 2024, 02:47 AM UTC
Work is still being performed in the IBM data center. All systems have remained stable since the last update and are expected to remain stable at this time. We will continue to monitor the status of this and will provide another update when it is available.
- monitoring Dec 11, 2024, 01:41 PM UTC
Update: As of 2024/12/11 04:30 EST, corrective changes were completed by IBM technicians. Since that time, systems have been monitored throughout this event and no further failover conditions have taken place since the last one yesterday, at 2024/12/10 15:47:12. Jenzabar Cloud Infrastructure will continue to monitor the environment for the next 24 hours before marking this incident as resolved. If you have any questions, concerns, or issues to report, please report them through normal channels at this time. Jenzabar Cloud Infrastructure
- resolved Dec 12, 2024, 01:42 PM UTC
At this time, all hardware is in great standing in our North America East data center location and no further corrective action is needed within the cluster. Marking this incident as resolved.