Jitterbit experienced a major incident on September 6, 2024 affecting Production Cloud Agent Group and Sandbox Cloud Agent Group, lasting 13h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 06, 2024, 06:20 AM UTC
We are currently investigating this issue.
- investigating Sep 06, 2024, 06:52 AM UTC
We are continuing to investigate this issue.
- investigating Sep 06, 2024, 07:23 AM UTC
We are continuing to investigate this issue.
- investigating Sep 06, 2024, 07:53 AM UTC
We are continuing to investigate this issue.
- identified Sep 06, 2024, 08:21 AM UTC
The issue has been identified and a fix is being implemented.
- identified Sep 06, 2024, 08:47 AM UTC
We are continuing to work on a fix for this issue.
- identified Sep 06, 2024, 08:49 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Sep 06, 2024, 09:01 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 06, 2024, 07:29 PM UTC
Issue has been resolved.
- postmortem Sep 06, 2024, 07:29 PM UTC
### Root Cause Analysis We apologize for the service disruption. We appreciate your patience and understanding. We are fully committed to resolving this and minimizing further disruptions. **Issue:** The cloud agents scheduler service failed to restart after a scheduled upgrade, disrupting customer schedules during the outage. **Impact:** Major **Services Impacted:** Customer scheduled operations using Jitterbit Sandbox and Production Cloud Agent groups Location: APAC Cloud **Problem Description:** Jitterbit iPaas scheduler service utilizes an external library for handling timezone-related functions. This library, upon initialization, attempts to update its timezone data file. An incorrect value within this file caused the Scheduler service to fail during startup. **Timeline:** **9/6/2024 \(UTC\)** **04:00 -** The upgrade process began and the old servers were drain stopped from service **05:15 -** The process was finished and internal testing began and discovered the issue with the scheduler service **08:20** - Identified the issue and started to develop a workaround and then apply to the Cloud Agents **08:55** - Workaround applied and confirmed that issues was mitigated **Root Cause:** A syntax error was found in third-party timezone data file **Action:** _Immediate Action:_ * Implemented a work around by fixing the timezone data file and restarting the service * Document workaround in release notes for customers that may run into this issue with their private agent _Strategic Action:_ * A future agent version will not update the timezone data file automatically. * Correct syntax of timezone data file