Exalate experienced a major incident on November 21, 2023 affecting Exalate Console and Synchronisation node and 1 more component, lasting 7d 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 21, 2023, 11:02 AM UTC
We’ve detected an issue where a limited number of nodes are experiencing Java Heap Space errors, leading to node downtime. Despite various recovery attempts, some nodes remain affected. Impact on Service This issue mainly impacts nodes running version 5.7. Not all nodes are affected, but those with the issue may experience downtime. Current Actions Our engineering team is actively developing and testing a custom solution. We plan to first deploy this fix on a selected number of nodes to ensure stability. Updates and Communication We are committed to keeping you informed about the progress with developments. Stay tuned for further updates.
- identified Nov 21, 2023, 02:52 PM UTC
Service Status Update: A fix version is on validation stage and we expect a hotfix to address the Java Heap Space error release anytime now. No action is required from your end, we will keep you updated as soon as this is deployed. We apologize for any inconvenience caused and appreciate your patience.
- identified Nov 21, 2023, 08:09 PM UTC
The deployment of the hotfix for the Java Heap error is expected to happen tonight at 22:00 CET for Jira cloud based nodes, other nodes will follow suit afterwards. Expect more updates shortly.
- identified Nov 21, 2023, 09:33 PM UTC
Hotfix release started for Jira Cloud based nodes, ETA 6-8 hours from now. Other nodes will follow suit afterwards.
- monitoring Nov 22, 2023, 07:00 AM UTC
Hotfix release completed for Jira Cloud based nodes. Other nodes will follow shortly.
- monitoring Nov 22, 2023, 01:24 PM UTC
Scheduled Hotfix: Zendesk & Github Connectors The hotfix for Zendesk and Github connectors is scheduled for 17:00 pm CET. We appreciate your understanding and patience as we work swiftly to enhance system stability.
- monitoring Nov 22, 2023, 05:03 PM UTC
The scheduled hotfix deployment for Zendesk and Github connectors has been postponed until further notice due to a database instability issue. We will keep you updated on any developments. Thank you for your comprehension.
- monitoring Nov 23, 2023, 02:11 PM UTC
We are currently working on a build, that encompasses the filtering of duplicates in sync events/requests and twin traces, along with the removal of duplicates during the upgrade. The build is currently undergoing internal testing for all cloud nodes.
- monitoring Nov 24, 2023, 06:46 AM UTC
The fix is on validation stage and will start deploying on the nodes shortly. Your node will be updated during the day. We are committed to keeping you informed about the progress with developments. Stay tuned for the resolution.
- monitoring Nov 27, 2023, 09:22 AM UTC
Deployment Status Update: Exalate for JCloud 5.7.2 The deployment of Exalate for JCloud 5.7.2 has been successfully completed. However, certain nodes are currently experiencing downtime as our team conducts essential duplicate trace clean-up. We are actively working to restore full functionality and appreciate your patience. Additionally, during this process, we have identified additional risks. To address these, a release (5.7.3) is planned and it is being tested, with the first deployment targeted for Jira connectors. We will keep you updated on our progress.
- resolved Nov 28, 2023, 04:37 PM UTC
This incident has been resolved.
- postmortem Apr 01, 2026, 04:15 AM UTC
timeout