Octopus incident

Dynamic Worker Leases failing in East AU

Major Resolved View vendor source →

Octopus experienced a major incident on April 28, 2024 affecting Octopus Cloud, lasting 14d 7h. The incident has been resolved; the full update timeline is below.

Started
Apr 28, 2024, 11:56 PM UTC
Resolved
May 13, 2024, 07:34 AM UTC
Duration
14d 7h
Detected by Pingoru
Apr 28, 2024, 11:56 PM UTC

Affected components

Octopus Cloud

Update timeline

  1. investigating Apr 28, 2024, 11:56 PM UTC

    Octopus Cloud is experiencing issues leasing Dynamic Workers in Australia East. The issue is currently under investigation and this incident will be updated when further details are known.

  2. investigating Apr 29, 2024, 12:52 AM UTC

    Our upstream provider, Microsoft Azure, is currently experiencing an issue which is affecting our ability to provision worker virtual machines for customers in East AU. We are working with Azure to resolve the issue. If you experience this problem, deployments utilizing dynamic workers will fail with an error specifying that Octopus Deploy could not obtain a dynamic worker lease. We will resolve this incident once we have received confirmation the issue has been resolved from our upstream provider.

  3. monitoring Apr 29, 2024, 01:15 AM UTC

    Dynamic Workers are now provisioning successfully in East AU. We are continuing to monitor for any degradation of service.

  4. monitoring Apr 29, 2024, 03:56 AM UTC

    We are continuing to see periods of service degradation in the Australia East region. New Dynamic Workers will be provisioned in the Australia Southeast region until we have confirmation from our service provider that the issue has been resolved.

  5. monitoring May 13, 2024, 07:28 AM UTC

    Our service provider, Microsoft Azure, has confirmed that the underlying issue has been resolved. We are continuing to monitor for any degradation of service.

  6. monitoring May 13, 2024, 07:32 AM UTC

    We have resumed provisioning Dynamic Workers in East AU (from Australia Southeast). We are continuing to monitor for any degradation of service.

  7. resolved May 13, 2024, 07:34 AM UTC

    This incident has been resolved.