Nutanix incident

Google Cloud Platform - intermittent issues starting VMs

Minor Resolved View vendor source →

Nutanix experienced a minor incident on February 3, 2022, lasting 7h 39m. The incident has been resolved; the full update timeline is below.

Started
Feb 03, 2022, 05:53 PM UTC
Resolved
Feb 04, 2022, 01:33 AM UTC
Duration
7h 39m
Detected by Pingoru
Feb 03, 2022, 05:53 PM UTC

Update timeline

  1. investigating Feb 03, 2022, 05:53 PM UTC

    We are seeing cases where VM starts are failing on Google Cloud Platform for a limited percentage of VMs. We are investigating, but at this point in time the issue appears to be limited to only some GCP regions. If you are experiencing any issues with your accounts using GCP, please stay tuned for updates here.

  2. identified Feb 04, 2022, 12:54 AM UTC

    We have determined that this issue is a result of a change in GCP behavior for VM starts that coincide with setting a label for the VM. A workaround is in development to accommodate this change and we expect to deploy this on Feb 4. We will update this incident once the workaround has been deployed and confirmed.

  3. resolved Feb 04, 2022, 04:33 AM UTC

    Those accounts affected by this issue have been manually addressed earlier today. Any future occurrences will be mitigated with the upcoming update, so we will close this incident. In case of any new issues related to GCP VM starts please report via a Nutanix Support ticket.

  4. postmortem Feb 04, 2022, 08:26 AM UTC

    This issue is now resolved as we have deployed and confirmed the workaround to the change in GCP behavior.