Edit

Share via

Cluster autoscaler fails to scale with "cannot scale cluster autoscaler enabled node pool" error

This article discusses how to resolve the "cannot scale cluster autoscaler enabled node pool" error that appears when scaling a cluster with an autoscaler enabled node pool.

Symptoms

You receive an error message that resembles the following message:

kubectl get nodes outputs "No resources found"
All pods state is Pending
Scale operations are failing with "Cannot scale cluster autoscaler enabled node pool" error

Troubleshooting checklist

Azure Kubernetes Service (AKS) uses virtual machine scale sets-based agent pools, which contain cluster nodes and cluster autoscaling capabilities if enabled.

Check that the cluster virtual machine scale set exists

  1. Sign in to Azure portal.

  2. Find the node resource group by searching the following names:

    • The default name MC_{AksResourceGroupName}_{YourAksClusterName}_{AksResourceLocation}.
    • The custom name (if it was provided at creation).

    Note

    When you create a new cluster, AKS automatically creates a second resource group to store the AKS resources. For more information, see Why are two resource groups created with AKS?

  3. Check the list of resources and make sure that there's a virtual machine scale set.

Cause 1: The cluster virtual machine scale set was deleted

Deleting the virtual machine scale set attached to the cluster causes the cluster autoscaler to fail. It also causes issues when provisioning resources such as nodes and pods.

Note

Modifying any resource under the node resource group in the AKS cluster is an unsupported action and will cause cluster operation failures. You can prevent changes from being made to the node resource group by blocking users from modifying resources managed by the AKS cluster.

Reconcile node pool

If the cluster virtual machine scale set is accidentally deleted, you can reconcile the node pool by using az aks nodepool update:

# Update Node Pool Configuration
az aks nodepool update --resource-group <resource-group-name> --cluster-name <cluster-name> --name <nodepool-name> --tags <tags> --node-taints <taints> --labels <labels>

# Verify the Update
az aks nodepool show --resource-group <resource-group-name> --cluster-name <cluster-name> --name <nodepool-name>

Monitor the node pool to make sure that it's functioning as expected and that all nodes are operational.

Cause 2: Tags or any other properties were modified from the node resource group

You may receive scaling errors if you modify or delete Azure-created tags and other resource properties in the node resource group. For more information, see Can I modify tags and other properties of the AKS resources in the node resource group?

Reconcile node resource group tags

Use the Azure CLI to make sure that the node resource group has the correct tags for AKS name and the AKS group name:

# Add or update tags for AKS name and AKS group name
az group update --name <node-resource-group-name> --set tags.AKS-Managed-Cluster-Name=<aks-managed-cluster-name> tags.AKS-Managed-Cluster-RG=<aks-managed-cluster-rg>

# Verify the tags
az group show --name <node-resource-group-name> --query "tags"

Monitor the resource group to make sure that the tags are correctly applied and that the resource group is functioning as expected.

Cause 3: The cluster node resource group was deleted

Deleting the cluster node resource group causes issues when provisioning the infrastructure resources required by the cluster, which causes the cluster autoscaler to fail.

Solution: Update the cluster to the goal state without changing the configuration

To resolve this issue, you can run the following command to recover the deleted virtual machine scale set or any tags (missing or modified):

Note

It might take a few minutes until the operation completes.

Set your environment variables for the AKS cluster resource group and cluster name before running the command. A random suffix is included to prevent name collisions during repeatable executions, but you must ensure the resource group and cluster exist.

export RANDOM_SUFFIX=$(head -c 3 /dev/urandom | xxd -p)
export AKS_RG_NAME="MyAksResourceGroup$RANDOM_SUFFIX"
export AKS_CLUSTER_NAME="MyAksCluster$RANDOM_SUFFIX"
az aks update --resource-group $AKS_RG_NAME --name $AKS_CLUSTER_NAME --no-wait

Contact us for help

If you have questions, you can ask Azure community support. You can also submit product feedback to Azure feedback community.

  • Last updated on