Camunda Outage History

Earlier today, starting around 10:20 AM EDT (14:20 UTC), Web Modeler SaaS became unresponsive for a short period after the release of version 8.9.31. The issue was identified as a Flyway migration-related problem, and we mitigated it by rolling back to version 8.9.30.

The issue has been identified as AWS incident in availability zone use1-az4/us-east-1b. We are working to mitigate the issue and will provide an update in the next 2hrs. Affected Region: AWS - US-EAST-1 Summary of AWS incident - https://health.aws.amazon.com/health/status AWS is currently experiencing an infrastructure issue in one Availability Zone of the US-EAST-1 region, caused by elevated temperatures in a single data center. As a result, some compute and storage resources in that zone are operating with reduced availability or increased latency. Other zones in US-EAST-1 are unaffected. AWS reports early signs of recovery: additional cooling capacity has been brought online and affected hardware is being restored in a controlled manner. Some workloads in the impacted zone may continue to see elevated error rates until full recovery is complete. We are actively monitoring the situation and will share further updates as the situation evolves.

The issue has been identified. We are working to mitigate the issue and will provide an update in the next 30min. Affected Versions: All C8 SaaS Affected Regions: - Belgium, Europe (europe-west1) - South Carolina, North America, (us-east1)

We are aware of a network issue affected the whole Camunda SaaS platform. We are working to mitigate the issue and will provide an update in the next [30min].

Camunda SaaS is currently impacted by an on-going issue with our service provider Cloudflare. Our team is working to mitigate the issue and will provide an update within the next hour.

Camunda Clusters running version 8.8 with app-integrations enabled are currently experiencing a service disruption affecting Connectors; when using this early access alpha feature. We’ve identified the root cause as a malformed configuration introduced in this version. A software fix has been developed and is currently undergoing testing in our integration environments.

We’ve identified an issue with the health evaluation logic in our Control Plane and are actively working on a fix. In the meantime, we’re manually triaging affected clusters to move them out of the “Update” state. The Camunda cluster itself is available and not affected from this.

We have developed a fix addressing the Control Plane health logic issue. The update is currently being tested and validated in our development and staging environments to ensure stability before production release. We’ve also completed triage of all Enterprise clusters to ensure the issue is not affecting the interactions in our SaaS Console.

We are aware of an issue impacting the AWS `us-east-1` region and we are closely monitoring the situation. See: https://health.aws.amazon.com/health/status We will update this incident as the situation progresses and will post an update within the next 1 hour. Affected Regions: * AWS: North America, Virginia (us-east-1)

We are continuing to work on a fix for this issue.

Our teams are still trying to mitigate the situation with our cloud provider. We are seeing a slow improvement of the requests made to AWS and are slowly getting back the missing compute capacity to run all our customers' workloads.

On all GCP regions, C8 workloads cannot start due to images not being pulled. As a result, pods cannot be started and C8 SaaS clusters can be in Unhealthy status. This affects different customers. Our team is working to identify the cause and implement a solution. Affected Regions: South Carolina, North America (us-east1), Iowa, North America (us-central1), Toronto, North America (northamerica-northeast2), Belgium, Europe (europe-west1), London, Europe (europe-west2), Sydney, Australia (australia-southeast1)

We are currently investigating reports of instances of IDP process extractions initiated on web-modeler that never complete. Our team is working to identify the cause and implement a solution. Affected Versions: Camunda 8.7.2 SaaS Affected Regions: Belgium, Europe (europe-west1), London, Europe (europe-west2) Franfkurt, Europe (eu-central-1) South Carolina, North America (us-east1) Iowa, North America (us-central1) Toronto, North America (northamerica-northeast2) Virgnia, North America (us-east-1) Singapore, Asia (asia-southeast1) Sydney, Australia (australia-southeast1)

We are currently investigating Zeebe outage for some clusters. Our team is working to identify the cause and implement a solution. Affected Regions: Iowa, North America (us-central1) & London, Europe (europe-west2)

We are currently investigating the problem with our cloud provider support engineers.

We are continuing our investigation of Zeebe and Elasticsearch outage for some clusters. In some cases, disks cannot be mounted. This can lead to situations, where either Zeebe processing, or data imported into WebApps (Operate, Tasklist, Optimize), or both, are impacted. Affected Regions: Iowa, North America (us-central1), Belgium, Europe (europe-west1), London, Europe (europe-west2), South Carolina, North America (us-east1), Toronto, North America (northamerica-northeast2), Toronto, North America (northamerica-northeast2), Sydney, Australia (australia-southeast1)

We are working closely with our cloud infrastructure provider to identify the cause of the issue. In some cases, disks cannot be mounted. This can lead to situations where either Zeebe processing, data imported into WebApps (Operate, Tasklist, Optimize), or both are impacted. Connectors might also be impacted due to transitivity with other affected components. Affected Regions: Iowa, North America (us-central1), Belgium, Europe (europe-west1), London, Europe (europe-west2), South Carolina, North America (us-east1), Toronto, North America (northamerica-northeast2), Toronto, North America (northamerica-northeast2), Sydney, Australia (australia-southeast1)

Our teams continue working closely with our cloud infrastructure provider to identify the cause of the issue and implement a solution

Our teams are actively implementing mitigations while continuing to investigate the root cause in collaboration with our cloud infrastructure provider.

We are observing improvements across all regions and we continue to closely monitor the situation to ensure full stability.

We have identified an issue with the Tasklist update from 8.3.x to 8.4.20 and 8.4.x to 8.5.15. After updating, the Tasklist importer will get stuck, leading to no new data being imported for Tasklist. General processing is *not* affected. We have the issue identified and are working on a fix asap. We have temporarily disabled updates in SaaS.

We disabled the update path from 8.6.13 to 8.7.0 because of a bug we identified after the update, that led to stopped Zeebe processing in some situations where multi-instances were in use. We are preparing an 8.7.1 patch, which will allow full recovery of affected clusters. No risk of data loss exists. Affected Versions: 8.6.13 to 8.7.0 update

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

We have identified an issue with Camunda 8.7-alpha3 on SaaS. This issue leads to data not appearing in Operate and Tasklist. We have disabled the creation of new alpha3 clusters. We have identified the root cause and are working on the solution. We expect the solution to be rolled out on Thursday (16/01). Once the solution has been rolled out, we will re-enable the creation of alpha3 SaaS clusters. Existing alpha3 will be reset once the solution is rolled out. Processes have to be re-deployed. Affected Versions: Camunda 8.7-alpha3 SaaS Affected Regions: Global

We have a solution in test and expect it to be rolled out on Friday (17/01). Once the solution has been rolled out, we will re-enable the creation of alpha3 SaaS clusters. Existing alpha3 will be reset once the solution is rolled out. Processes have to be re-deployed. Affected Versions: Camunda 8.7-alpha3 SaaS Affected Regions: Global

We have identified an issue where a small number of Optimize instances were not starting between 12:00 and 17:00. This only impacted new Optimize 8.6.1 instances. No other components are affected and we have not identified any data corruption. We apologize for any inconvenience caused and thank you for your patience, and have already applied a fix for impacted clusters.

An issue has been identified as a critical bug in the 8.6+Gen1 generation in Zeebe. This can potentially lead to data corruption. We have not detected any data corruption yet and we consider the likelihood of a data corruption low, but we want to proactively prevent this through a hotfix. We have disabled the creation of new clusters using this generation until the hotfix is available. We are working to mitigate the issue and will provide an update in the next 1hr. Affected Versions: 8.6+Gen1 Affected Regions: Belgium, Europe (europe-west1), London, Europe (europe-west2), South Carolina, North America (us-east1), Iowa, North America (us-central1), Toronto, North America (northamerica-northeast2), Sydney, Australia (australia-southeast1)

We are continuing to work on a fix for this issue.

We are preparing a new release fixing the issue.