Exalate incident

Jira Cloud IssueLinks being Removed

Minor Resolved View vendor source →

Exalate experienced a minor incident on August 29, 2025 affecting Synchronisation node, lasting 6d 19h. The incident has been resolved; the full update timeline is below.

Started
Aug 29, 2025, 04:18 PM UTC
Resolved
Sep 05, 2025, 11:56 AM UTC
Duration
6d 19h
Detected by Pingoru
Aug 29, 2025, 04:18 PM UTC

Affected components

Synchronisation node

Update timeline

  1. investigating Aug 29, 2025, 04:18 PM UTC

    We are investigating a problem where IssueLinks are being removed from Jira cloud tickets. Progress and actions will be notified accordingly, if you encountered this problem please reach us on support portal.

  2. identified Aug 29, 2025, 04:57 PM UTC

    The issue has been identified and we are performing remediation actions.

  3. identified Aug 29, 2025, 09:04 PM UTC

    All nodes have been moved from the affected version and a workaround to remediate the possible effects on issue links is being investigated.

  4. monitoring Aug 31, 2025, 07:00 PM UTC

    We identified the affected instances and provided remediation steps to all affected clients to restore the removed links.

  5. resolved Sep 05, 2025, 11:56 AM UTC

    The issue affecting Jira Cloud issue links has been fully resolved. A detailed postmortem will be published soon outlining the root cause and preventive measures being implemented.

  6. postmortem Sep 18, 2025, 06:57 AM UTC

    # Exalate v5.28 Issue Links Synchronization - Technical Incident Report **Incident Date:** 29/Aug/2025 **Duration:** ~4 hours from detection to rollback completion **Impact:** Issue link synchronization behavior change affecting multiple customer instances **Status:** Resolved with data restoration completed ## Timeline The incident was first detected on 29/Aug at 17:56 when a customer reported issue links being removed after upgrading to v5.28. Within eight minutes, at 18:04, the support team received multiple similar reports and immediately requested a rollback. The formal incident response was initiated at 18:17, and the rollback process to v5.27.0 began at 18:48. The rollback was completed across all affected instances by 22:12 the same day, approximately four hours after initial detection. Recovery efforts continued over the weekend. On 30/Aug at 09:43, the data restoration team was assembled and began developing recovery procedures. The comprehensive customer communication deployment occurred on 01/Sep at 17:05, providing affected customers with restoration instructions and individualized support. _\(All times CET\)_ ‌ ‌ ## Root Cause Analysis The technical issue stemmed from a synchronization enhancement in v5.28 that compared issue links between source and destination systems. The implementation operated under the assumption that links present on the destination but not on the source represented synchronization inconsistencies requiring removal. This approach failed to distinguish between links managed by Exalate and links created independently outside the synchronization scope. Several contributing factors enabled this issue to reach production. The scope definition did not account for mixed environments where synchronized and non-synchronized links coexist on the same system. Test coverage focused primarily on standard synchronization flows rather than these mixed data environments. Additionally, the modification affected a broader scope than initially anticipated during the development and review process. The impact characteristics were significant but recoverable. Multiple customer instances experienced removal of issue links that existed outside the intended synchronization scope. However, service functionality continued throughout the incident, and comprehensive audit logs preserved all affected link data, which enabled complete restoration of the removed links.Regarding data security and privacy, the audit logs contain only metadata elements such as issue identifiers, link types, and relationship mappings. No sensitive customer content, comments, or detailed issue information was captured in these logs. This metadata-only approach ensured that the restoration process could proceed while maintaining appropriate data protection standards. ## Learnings On the positive side, the incident response team made a rapid rollback decision, quickly identifying this as the appropriate technical solution rather than attempting a forward fix. The comprehensive logging infrastructure proved invaluable, enabling complete data restoration from audit trails. Cross-functional coordination worked well across engineering, customer success, and operations teams. The structured communication approach maintained clarity and transparency throughout the resolution process. The incident also revealed important areas for system enhancement. Synchronization logic in complex integration scenarios requires better handling of mixed data environments where multiple systems manage different subsets of the same data types. Testing scope needs expansion to cover mixed synchronized and non-synchronized environments comprehensively. Change evaluation processes require enhancement for modifications that affect customer data. Monitoring capabilities need improvement to enable proactive detection of synchronization behavior changes before customer impact. Several process improvements have been \(or are being\) implemented as a result. Enhanced code review procedures now include additional requirements specifically for data modification operations. Expanded testing protocols cover scenarios involving mixed data environments and include dedicated data preservation validation. Improved monitoring systems detect synchronization pattern changes and unusual data modification activities. Updated release procedures include enhanced evaluation criteria for data-affecting changes and improved rollback automation capabilities.