Octopus incident

Dynamic Worker Leases failing in WestEU

Major Resolved View vendor source →

Octopus experienced a major incident on January 21, 2024 affecting Octopus Cloud, lasting 16h 37m. The incident has been resolved; the full update timeline is below.

Started
Jan 21, 2024, 06:31 AM UTC
Resolved
Jan 21, 2024, 11:09 PM UTC
Duration
16h 37m
Detected by Pingoru
Jan 21, 2024, 06:31 AM UTC

Affected components

Octopus Cloud

Update timeline

  1. identified Jan 21, 2024, 06:31 AM UTC

    Our upstream provider, Microsoft Azure, is currently experiencing an issue which is affecting our ability to provision worker virtual machines for a subset of customers in WestEU. See https://azure.status.microsoft/en-gb/status for more details. If you experience this problem, deployments utilising dynamic workers will fail with an error specifying that Octopus Deploy could not obtain a dynamic worker lease. We will provide an update in an hour, at approximately 0730 UTC. We will resolve this incident once we have received confirmation the issue has been resolved from our upstream provider.

  2. identified Jan 21, 2024, 07:31 AM UTC

    We continue to see issues with our upstream. We are investigating a workaround. Another update will be provided within the next hour.

  3. monitoring Jan 21, 2024, 09:17 AM UTC

    Dynamic Workers are now provisioning successfully. We are continuing to monitor for any degradation of service.

  4. resolved Jan 21, 2024, 11:09 PM UTC

    Azure has advised that this issue has been resolved. A preliminary root cause has been published here: https://azure.status.microsoft/en-us/status/history/