Octopus Outage History

Started Jan 30, 2025, 07:24 AM UTC· resolved Feb 04, 2025, 03:07 AM UTC

1. investigating Jan 30, 2025, 07:24 AM UTC

   Internal monitoring notified us that a small number of Octopus Cloud instances were terminated for excess memory use up to 12 hours after a recent a software upgrade. These instances were automatically restarted and resumed functioning correctly. At this stage we are investigating to determine the cause of the excess memory use. If you believe you are affected by this issue please contact [email protected]

2. resolved Feb 04, 2025, 03:07 AM UTC

   Two seperate recent changes were identified as causes of the excess memory use. These changes have been rolled back, and we have confirmed through monitoring that memory use has returned to normal on the affected instances. No further crashes terminations have been observed.

Read the full incident report →

Started Jan 24, 2025, 01:35 AM UTC· resolved Jan 27, 2025, 10:21 PM UTC

1. identified Jan 24, 2025, 01:35 AM UTC

   We have had reports of and have identified an issue with variable substitution with Helm template sources. We are tracking the fix here: https://github.com/OctopusDeploy/Issues/issues/9224

2. monitoring Jan 24, 2025, 08:58 AM UTC

   We have fixed the issue, but due to timing and other factors will not be rolling this out immediately. If you are affected by this issue, please get in touch with [email protected] who can manually update your instance.

3. resolved Jan 27, 2025, 10:21 PM UTC

   This issue has been resolved and will be rolling out to Octopus Cloud customers in the next couple of days. If you are affected by this issue, please get in touch with [email protected]

4. postmortem Feb 05, 2025, 10:30 PM UTC

   # Report and learnings: Missing variable substitution on Helm template sources ###### Author: Kevin Tchang # Summary The Octopus Server `2025.1.6849` update introduced a bug in the _Deploy a Helm Chart_ step, preventing variable substitution from working correctly in certain Helm template value sources. This led to Helm deployment failures for Cloud customers, which had previously succeeded without issue. # Background [Calamari](https://octopus.com/docs/octopus-rest-api/calamari) is a deployment tool used by Octopus to execute deployment tasks on target machines, such as extracting packages and running scripts. It supports the deployment of many built-in Octopus steps, including the _Deploy a Helm Chart_ step, which allows users to deploy Helm charts to Kubernetes clusters. When deploying Helm charts, Octopus allows users to pass values into the Helm release via **Helm Template Value Sources** \(Helm TVS\). These values can come from various sources, including charts, packages, Git repositories, key-values, or inline YAML. The Helm TVS are sent to Calamari during deployment to configure the Helm chart correctly. **Octopus variable substitution** allows dynamic replacement of placeholder variables within deployment configurations. This enables users to store sensitive data or environment-specific values as variables in Octopus and substitute them during deployment. ## Incident timeline _\(All dates and times below are shown in UTC\)_ ##### 23/1/2025 – 18:49 \(5:49 AEDT\) We receive the first reports of Helm deployments failing due to missing variable substitutions. ##### 23/1/2025 – 23:39 \(10:39 AEDT\) Our support team escalated the issue to our engineering teams. ##### 24/1/2025 – 1:35 \(12:35 AEDT\) Our internal incident response process was initiated. ##### 24/1/2025 – 8:58 \(19:58 AEDT\) The fix for the bug is merged `2025.1.7389`, and our Status Page is updated to _Monitoring_. However, due to the timing of a recent Octopus Server upgrade to .NET 9 and concerns about stability, the fix was not immediately rolled out to all Cloud customers. In the meantime, our support team provided assistance with manual upgrades. ##### 27/1/2025 – 22:21 \(9:21 AEDT\) After the long weekend \(due to a public holiday\), the Status Page was updated to _Resolved_ once the fix was confirmed by customers. ## Technical details Helm TVS are transmitted to Calamari as strings within a JSON array structure. Upon receiving these, Calamari parses the JSON array into their respective TVS types \(such as package, chart, inline-YAML, etc.\), which are then used by the `helm upgrade` command. The bug was inadvertently introduced during a recent change aimed at eliminating the need for escaping quotes when using Octopus variable values in Helm TVS. Prior to this change, variable values with unescaped quotes caused parsing errors because the quotes were misinterpreted when processed from the JSON array structure. The change modified the process so that variable substitution occurs after the JSON array is parsed into its respective TVS types, rather than before. However, this substitution was initially applied only to the **inline YAML TVS type**, and not the other four TVS types, which also could contain variables to substitute. This resulted in a regression, and the fix involved applying variable substitution to all TVS types and ensuring it was handled correctly wherever a variable could appear. ## Remediation and next steps At Octopus, ensuring deployment reliability is a top priority. Following this incident, we conducted a comprehensive review to identify areas for improvement. Given the flexibility and complexity of our product features, designing a robust testing process for all scenarios can be challenging. However, we’re taking proactive steps to improve how we test variable evaluation features.

Read the full incident report →

Started Jan 22, 2025, 05:47 PM UTC· resolved Jan 23, 2025, 07:49 AM UTC

1. investigating Jan 22, 2025, 05:47 PM UTC

   We are currently investigating the issue

2. identified Jan 23, 2025, 01:42 AM UTC

   We have identified the issue with a recently released version of Octopus Server to cloud customers. A recent change has caused config maps with multi-line variables, created via the Kubernetes containers deployment step, to fail. Config maps created by the dedicated Kubernetes config map step are not affected, nor config maps created by Raw YAML or Helm steps. We are in the process of fixing this and will update the public issue with the fixed version: https://github.com/OctopusDeploy/Issues/issues/9221.

3. monitoring Jan 23, 2025, 02:47 AM UTC

   A fix has been implemented and will be rolled out to affected cloud customers in their next maintenance window.

4. resolved Jan 23, 2025, 07:49 AM UTC

   We are currently rolling out the fix to Cloud customers. Instances will be upgraded in their next maintenance window. If you are affected by this issue and want to expedite the upgrade, please contact [email protected]

5. postmortem Feb 05, 2025, 10:04 PM UTC

   # Report and learnings: Configure and apply Kubernetes resources step error parsing configmap.yml ###### Author: Kevin Tchang ## Summary In Octopus Server `2025.1.5751`, a bug caused the deployment of Kubernetes config maps containing multi-line variables, when created through the _Configure and apply Kubernetes resources_ step \(built-in Kubernetes step for deploying containers\), to fail. Config maps created using the dedicated Kubernetes config map step, as well as those generated with Raw YAML or Helm steps, were unaffected. This issue impacted Cloud customers, who experienced deployment failures that had previously been successful. The bug was a regression caused by a change supporting manifest reporting for Kubernetes deployment steps, part of an upcoming feature. This change mistakenly caused line breaks in multi-line Octopus variable values to not be properly escaped when substituted into the config map's key-value pairs. The problem became apparent when customers had PEM certificates or JSON blobs that needed to be inserted into the config map. These were replaced verbatim in Calamari, leading to YAML formatting issues due to unescaped line breaks. ## Background The [_Configure and apply Kubernetes resources_](https://octopus.com/docs/kubernetes/steps/kubernetes-resources) step deploys a combination of Kubernetes Deployment, Service, and Ingress resources. It also allows the optional configuration and deployment of an associated Kubernetes ConfigMap and Secret for reference by the Deployment. To support Rolling Update and Blue/Green deployment strategies, ConfigMap and Secret resources must have unique names for each Deployment version. These resources are assigned [computed names](https://octopus.com/docs/kubernetes/steps/kubernetes-resources?q=configmap#configmap-and-secret), which, by default, combine the resource name with the Octopus deployment ID, and are determined only at deployment time. ## Incident timeline _\(All dates and times below are shown in UTC\)_ ##### **22/1/2025 – 7:31 \(18:31 AEDT\)** Began receiving customer reports of an increase in failing Kubernetes deployments. These failures have been observed across various projects, with similar errors related to parsing config maps. Our support team worked with our customers to troubleshoot the reasons for the failures. ##### **22/1/2025 – 10:58 Jan 22, 2025 \(21:58 AEDT\)** Our support team escalated the issue to our engineering teams. ##### **22/1/2025 – 15:51 \(2:51 AEDT\)** Our internal incident response process was initiated. ##### **22/1/2025 – 21:45 \(8:45 AEDT\)** Our engineers logged on and begin to identify the cause of the incident. ##### **23/1/2025 – 1:42 \(12:42 AEDT\)** The fix for the bug is merged, and our Status Page is updated to _Identified._ ##### **23/1/2025 – 2:47 \(13:47 AEDT\)** Our Status Page is updated to _Monitoring_ as we begin the process to expedite the release `2025.1.7128` of the fix to our affected Cloud customers. ##### **23/1/2025 – 7:49 \(18:49 AEDT\)** Status Page updated to _Resolved._ ## Technical details Before the change to support manifest reporting, the Kubernetes container deployment step created associated Kubernetes config maps \(and secrets\) using the `kubectl create` command with the `--from-files` flag, where each config map key-value pair was sent to Calamari as an individual file. This process was updated to use the more standard `kubectl apply -f` method, where Octopus now sends a single YAML manifest to Calamari representing the config map. The YAML is generated from a config map resource that we build as an in-memory C# object. The bug was introduced when the argument for the config map object used raw, unevaluated Octopus variable values. The issue wasn’t identified during testing because the deployment step involves two stages of variable substitution: the first on Octopus Server, and the second inside Calamari during deployment. The two substitution passes are necessary to support the use of computed names, ensuring that each deployment version has its own unique resources. The change didn't account for multi-line strings as potential variables, causing newline characters to not be properly escaped before serialization. This issue occurred because encoding needs to happen on Octopus Server before the object is serialized into YAML. The second substitution in Calamari is direct on the YAML file. The bug was a regression, and the fix involved evaluating the values before serialization to ensure newline characters were handled correctly. ## Remediation and next steps At Octopus, we take deployment reliability very seriously. After this incident, we conducted a thorough review to identify areas where we can improve our processes, in light of the lessons learned. We’ve identified a complex and unconventional area of the code—specifically script-based Kubernetes deployments—that requires further attention. Given the distinctive challenges these deployments present, we are committed to enhancing this area with additional tests to ensure better reliability.

Read the full incident report →

Started Jan 14, 2025, 12:12 AM UTC· resolved Jan 14, 2025, 12:38 AM UTC

1. identified Jan 14, 2025, 12:12 AM UTC

   For customers using Github to store Config As Code projects, an ongoing incident (https://www.githubstatus.com/incidents/qd96yfgvmcf9) with Github is affecting the loading of those projects. There is no impact to projects that are using database storage.

2. resolved Jan 14, 2025, 12:38 AM UTC

   The Github incident has been resolved, we have confirmed that Config-as-Code project loading is working correctly

Read the full incident report →

Started Jan 07, 2025, 12:20 AM UTC· resolved Jan 07, 2025, 03:10 AM UTC

1. investigating Jan 07, 2025, 12:20 AM UTC

   We are currently investigating the issue.

2. identified Jan 07, 2025, 01:17 AM UTC

   14 Instances are down at this time. We've identified the issue and are working to restore services as a matter of priority.

3. monitoring Jan 07, 2025, 01:55 AM UTC

   Root cause has been identified and mitigated. All Instances are now running. Further updates will be provided shortly.

4. resolved Jan 07, 2025, 03:10 AM UTC

   All affected Instances are now up and healthy.

Read the full incident report →

Started Dec 09, 2024, 09:32 PM UTC· resolved Dec 11, 2024, 12:08 AM UTC

1. investigating Dec 09, 2024, 09:32 PM UTC

   Some Octopus Cloud customers are having intermittent issues with package acquisition steps throwing exceptions. We are currently investigating.

2. identified Dec 09, 2024, 11:56 PM UTC

   We are continuing to actively work on this issue. We have identified the issue and are testing a potential mitigation/fix. We are also continuing to investigate the underlying causes of the issue to build confidence our mitigation will work as expected.

3. identified Dec 10, 2024, 01:39 AM UTC

   We are continuing to actively work on resolving this issue. Our first potential mitigation didn't work as expected. We have a second potential mitigation that is performing better in testing. We will update here if/when we ship it.

4. identified Dec 10, 2024, 02:13 AM UTC

   We've identified the underlying code change that caused this issue to start appearing. We have a fix for it that will be rolling out to Cloud customers as quickly as possible. Self-hosted customers will not be affected.

5. monitoring Dec 10, 2024, 07:27 AM UTC

   The fix is now rolling out to Cloud. If this issue is affecting you, please reach out to the support team to ask them to upgrade your instance to the latest version.

6. resolved Dec 11, 2024, 12:08 AM UTC

   The fix has been released to Cloud - if you are still encountering issues, please ask the support team to upgrade your instance to 2025.1.3365

Read the full incident report →

Started Nov 20, 2024, 04:55 AM UTC· resolved Nov 20, 2024, 07:12 AM UTC

1. investigating Nov 20, 2024, 04:55 AM UTC

   We are currently experiencing an issue with Octopus Cloud Trial signups. New Trial instances are not being provisioned as expected. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. We will provide updates here as soon as more information becomes available. Users attempting to sign up for new Octopus Cloud Trials are unable to provision new trial instances at this time.

2. identified Nov 20, 2024, 06:50 AM UTC

   The cause has been identified and engineers are working on a fix

3. identified Nov 20, 2024, 06:53 AM UTC

   The cause of the issue has been identified and engineers are working on a fix

4. resolved Nov 20, 2024, 07:12 AM UTC

   This issue has now been resolved and all systems are working as expected

Read the full incident report →

Started Nov 20, 2024, 02:15 AM UTC· resolved Nov 20, 2024, 02:25 AM UTC

1. resolved Nov 20, 2024, 03:49 AM UTC

   Between 12:15 PM and 12:25 PM AEST, a brief outage affected Octopus ID and a subset of Octopus.com services, including the Profile, Control Centre v1, Blogs, and License Purchasing. During this time, customers were unable to log in to cloud instances using Octopus ID and access certain areas of the site. The issue was caused by an unintended outage during a system upgrade. The issue was identified and resolved promptly, restoring all services within 10 minutes. All impacted systems are fully operational, and no further issues are expected.

Read the full incident report →