Cloud instances unable to communicate with listening and polling tentacles

Update timeline

1. investigating Dec 08, 2023, 01:23 AM UTC

   Since the morning of Friday 8th December, cloud customers are reporting issues with communicating to listening and polling tentacles. We are investigating this as a priority incident and will work on a resolution as soon as possible.

2. investigating Dec 08, 2023, 03:41 AM UTC

   We have identified a suspected root cause and workaround. We suspect this is limited to certificates using a SHA1 algorithm but are investigating further. If affected please try the workaround detailed in this issue: https://github.com/OctopusDeploy/Issues/issues/8523

3. identified Dec 08, 2023, 06:06 AM UTC

   The underlying cause has been identified. The issue is currently isolated to cloud customers. A fix is actively being work on.

4. monitoring Dec 08, 2023, 10:21 AM UTC

   A fix has been created and is being rolled out to affected customers. This issue affects instances of Tentacle that use SHA1 (an old, unsupported, encryption algorithm), and prevents them from connecting to Octopus Cloud. This was a side effect of the Octopus Server .NET 8 upgrade, and the resulting change to the underlying OS from Debian 11/OpenSSL 1.x to Debian 12/OpenSSL 3.x. If you continue to be affected by this issue, please try the workaround detailed at https://github.com/OctopusDeploy/Issues/issues/8523, or contact support at https://octopus.com/support.

5. monitoring Dec 11, 2023, 01:31 AM UTC

   We have identified customers that were most likely affected. and prioritized applying the fix to these customers on the 9th and 10th of December. Rollout continues to all remaining customers, and we anticipate this to be complete by the 12th of December.

6. resolved Dec 13, 2023, 02:45 AM UTC

   All Octopus Cloud customers have been upgraded to fixed software versions, resolving this incident. Self-hosted customers were not affected.