Octopus incident

Cloud instances may be missing task log lines or show incorrect status for deployment steps

Minor Resolved View vendor source →

Octopus experienced a minor incident on December 6, 2023 affecting Octopus Cloud, lasting 13d 21h. The incident has been resolved; the full update timeline is below.

Started
Dec 06, 2023, 06:20 AM UTC
Resolved
Dec 20, 2023, 03:54 AM UTC
Duration
13d 21h
Detected by Pingoru
Dec 06, 2023, 06:20 AM UTC

Affected components

Octopus Cloud

Update timeline

  1. investigating Dec 06, 2023, 06:20 AM UTC

    We have identified an issue where some Octopus Cloud instances since 20 November 2023 are missing some log lines from their task logs. Deployment steps that encounter this problem could also show the wrong status (e.g. showing as still running) even though the step has completed successfully. The deployment task will have the correct status. We are investigating this issue as a priority.

  2. investigating Dec 07, 2023, 01:17 AM UTC

    We have identified a workaround. The issue occurs when a deployment's task log is viewed while the deployment is running. If users navigate away from viewing a running deployment, all task log lines should be successfully saved. This will also avoid the issue where deployment steps may show the wrong status. We are working with our upstream provider to resolve this issue.

  3. identified Dec 11, 2023, 12:26 AM UTC

    We have identified the cause of the issue and are testing mitigations. We are working on this issue as a priority.

  4. identified Dec 12, 2023, 06:41 AM UTC

    We are currently rolling out a fix for this issue. We will provide an update once all Octopus Cloud instances have the fix applied.

  5. monitoring Dec 15, 2023, 03:42 AM UTC

    A fix has been implemented on all Octopus Cloud instances and we are monitoring for any further errors.

  6. resolved Dec 20, 2023, 03:54 AM UTC

    This issue has been resolved and we have not seen any further errors.