Teradici incident

Cloud Access Connector upgrade failures

Minor Resolved View vendor source →

Teradici experienced a minor incident on November 19, 2020 affecting Anyware Manager, lasting 4d 16h. The incident has been resolved; the full update timeline is below.

Started
Nov 19, 2020, 01:17 AM UTC
Resolved
Nov 23, 2020, 06:07 PM UTC
Duration
4d 16h
Detected by Pingoru
Nov 19, 2020, 01:17 AM UTC

Affected components

Anyware Manager

Update timeline

  1. investigating Nov 19, 2020, 01:17 AM UTC

    Some customers are experiencing failures upgrading their Cloud Access Connectors. At this time, we recommend provisioning a new virtual machine and deploying a new Cloud Access Connector or postponing upgrading while we investigate this issue.

  2. monitoring Nov 19, 2020, 06:21 AM UTC

    Our investigation has linked the reported upgrade failures to a change that was introduced on November 5 which resulted in users receiving authorization errors during install or upgrade. The issue was resolved for installations and upgrades when using the most recent installer later that day and additional fixes were applied on November 13 which resolved the authorization errors during if an upgrade was attempted from a previous installer release. After the November 13 update, any user attempting upgrade using an installer downloaded prior to November 5 will receive an error message stating "The installer is out of date". This has led to some confusion due to the version listed in the filename. If you receive this message, obtain the latest installer as per the instruction, even if it has the same version number in the file name as the one you are currently using. Some users have also reported running out of disk space during upgrades. While this is not related to the issue introduced on November 5 and resolved on November 13, it is an issue that can occur during upgrades. During our investigation we identified a possible or contributing source of the problem of unused images left after upgrades. To resolve this, prior to upgrading run the command docker image prune -a or sudo docker image prune -a (depending on if your host is configured to require sudo to run docker commands). We are continuing to monitor the situation to ensure there are no remaining issues from the initial change on November 5 or the fixes on November 5 and November 13. As well, we are investigating improvements and solutions for the Cloud Access Connector to the versioning confusion and disk space exhaustion. We will provide an update during during the next business day. If you experience any issues during an upgrade of your Cloud Access Connector please open a support case at https://help.teradici.com/.

  3. monitoring Nov 19, 2020, 07:40 PM UTC

    We are continuing to monitor for any further issues. In addition to the previous information, any customers that block outbound Internet connections from their Cloud Access Connector may see connection failures or timeouts during install or upgrade. To resolve this add the following FQDNs to the proxy or firewall's allowlist rules: * dl.teradici.com * docker.cloudsmith.io The complete list of FQDNs to allow are available here: https://www.teradici.com/web-help/pcoip_cloud_access_manager/CACv2/overview/feature_support/

  4. monitoring Nov 21, 2020, 12:15 AM UTC

    We are continuing to monitor for any further issues. If any issues are identified over the weekend we will provide status updates. Otherwise the next status update will occur on Monday.

  5. monitoring Nov 21, 2020, 05:50 AM UTC

    We are continuing to monitor for any further issues.

  6. identified Nov 21, 2020, 05:58 AM UTC

    We are experiencing an interruption in the ability to install or update the Cloud Access Connector. While we work to resolve this issue please include the flag --docker-registry teradici-docker-registry.bintray.io in your install or upgrade command in order to be able to successfully complete the install or upgrade: * sudo ./cloud-access-connector install --docker-registry teradici-docker-registry.bintray.io .. * sudo ./cloud-access-connector update --docker-registry teradici-docker-registry.bintray.io ...

  7. identified Nov 21, 2020, 06:16 AM UTC

    We are experiencing an interruption in the ability to install or update the Cloud Access Connector. While we work to resolve this issue please include the flag --docker-registry teradici-docker-registry.bintray.io in your install or upgrade command in order to be able to successfully complete the install or upgrade: * sudo ./cloud-access-connector install --docker-registry teradici-docker-registry.bintray.io .. * sudo ./cloud-access-connector update --docker-registry teradici-docker-registry.bintray.io ... In addition, downloads for PCoIP Clients and PCoIP Agents for Windows are currently unavailable.

  8. identified Nov 21, 2020, 06:39 AM UTC

    We are continuing to work on this issue. Downloads for PCoIP Clients and Agents have been resolved. Install and upgrade of the Cloud Access Connector remains impacted. To install or upgrade the Cloud Access Connector please include the flag --docker-registry teradici-docker-registry.bintray.io in your install or upgrade command in order to be able to successfully complete the install or upgrade: * sudo ./cloud-access-connector install --docker-registry teradici-docker-registry.bintray.io .. * sudo ./cloud-access-connector update --docker-registry teradici-docker-registry.bintray.io ...

  9. monitoring Nov 21, 2020, 04:07 PM UTC

    We have resolved the issue impacting downloads of PCoIP Clients and Agents, and installs and upgrades of the Cloud Access Connector. We will continue to monitor the situation for any additional issues. If you experience any issues downloading PCoIP software, or installing or upgrading your Cloud Access Connector please open a support case at https://help.teradici.com.

  10. resolved Nov 23, 2020, 06:07 PM UTC

    This incident has been resolved.