Resource limits, VM sizes, and regions for AKS on Windows Server

Applies to: AKS on Windows Server

This article provides information about tested configurations, resource limits, VM sizes, and regions for Azure Kubernetes Service (AKS) on Windows Server. The tests use the latest release of AKS on Windows Server.

Maximum specifications

AKS on Windows Server deployments were validated with the following configurations, including the specified maximums. If you exceed these maximums, you might encounter unexpected behaviors and failures. This article provides guidance on how to avoid common configuration mistakes and can help you create a larger configuration. If you need assistance, contact your local Microsoft office or submit a question to the community.

The recommended limits were tested with the default virtual machine (VM) sizes, based on the following table:

The hardware configuration of each physical node in the AKS Arc cluster is as follows:

• Chassis: Dell PowerEdge R650 Server or similar.
• RAM: RDIMM, 3200 MT/s, Dual Rank, total of 256 GB.
• CPU: Two (2) Intel Xeon Silver 4316 2.3G, 20C/40T, 10.4 GT/s, 30M Cache, Turbo, HT (150 W) DDR4-2666.
• Disk: 8x HDDs (2 TB or larger) and 2x 1.6 TB NVMe to support S2D storage configurations.
• Network: Four (4) 100-Gbit NICs (Mellanox or Intel).

Microsoft engineering tested AKS on Windows Server using this configuration, for single node, two node, four node, and eight node Windows failover clusters. If you have a requirement that exceeds the tested configuration, see Scaling AKS.

Available VM sizes

The following VM sizes for control plane nodes, Linux worker nodes, and Windows worker nodes are available for AKS on Windows Server. While VM sizes such as Standard_K8S2_v1 and Standard_K8S_v1 are supported for testing and low resource requirement deployments, use these sizes with care and apply stringent testing as they might cause unexpected failures due to out of memory conditions.

Supported Azure regions for Azure registration

AKS on Windows Server is supported in the following Azure regions:

• Australia East
• East US
• Southeast Asia
• West Europe

Scaling AKS

Scaling an AKS deployment requires planning. You need to understand your workloads and your target cluster utilization. Also, consider the hardware resources in your underlying infrastructure, such as total CPU cores, total memory, storage, and IP addresses.

The following examples assume that you deploy only AKS-based workloads on the underlying infrastructure. If you deploy non-AKS workloads, such as standalone or clustered virtual machines or database servers, you reduce the resources available to AKS.

Before you start, consider the following points to determine your maximum scale and the number of target clusters you need to support:

• The number of IP addresses you have available for pods in a target cluster.
• The number of IP addresses available for Kubernetes services in a target cluster.
• The number of pods you need to run your workloads.

To determine the size of your Azure Kubernetes Service host VM, you need to know the number of worker nodes and target clusters. This information determines the size of the AKS host VM. For example:

• The number of target clusters in the final deployed system.
• The number of nodes, including control plane, load balancer, and worker nodes across all target clusters.

To determine the size of your target cluster control plane node, you need to know the number of pods, containers, and worker nodes you plan to deploy in each target cluster.

Default settings that you can't change in AKS

During or after deployment, you can't change some default configurations and settings. These settings can limit the scale for a given target cluster.

• The subnet 10.244.0.0/16 limits the number of IP addresses for Kubernetes pods in a target cluster. That limit means each worker node can use 255 IP addresses for pods across all Kubernetes namespaces. To see the pod CIDR assigned to each worker node in your cluster, use the following command in PowerShell:

kubectl get nodes -o json | findstr 'hostname podCIDR'

                    "kubernetes.io/hostname": "moc-lip6cotjt0f",
                                "f:podCIDR": {},
                                "f:podCIDRs": {
                                    "f:kubernetes.io/hostname": {},
                "podCIDR": "10.244.2.0/24",
                "podCIDRs": [
                    "kubernetes.io/hostname": "moc-lmb6zqozk4m",
                                "f:podCIDR": {},
                                "f:podCIDRs": {
                                    "f:kubernetes.io/hostname": {},
                "podCIDR": "10.244.4.0/24",
                "podCIDRs": [
                    "kubernetes.io/hostname": "moc-wgwhhijxtfv",
                                "f:podCIDR": {},
                                "f:podCIDRs": {
                                    "f:kubernetes.io/hostname": {},
                "podCIDR": "10.244.5.0/24",
                "podCIDRs": [

In the example, you see three nodes with three CIDRs, and each node can host 254 pods. For performance reasons, the Kubernetes scale documentation recommends that you don't exceed 110 pods per node. See Considerations for large clusters.

Other considerations:

• The 10.96.0.0/16 address pool provides IP addresses for Kubernetes services outside the VIP pool you allocated. The system uses one of the 255 available addresses for the Kubernetes API server.
• You can only set the size of the AKS host VM during installation when you run Set-AksHciConfig for the first time. You can't change it later.
• You can only set the size of target cluster control plane and load balancer VMs when you create the target cluster.

Scale example

The following scaling example is based on these general assumptions and use cases:

• You want to tolerate the loss of one physical node in the Kubernetes cluster.
• You want to support upgrading target clusters to newer versions.
• You want high availability for the target cluster control plane nodes and load balancer nodes.
• You want to reserve a part of the overall Windows Server capacity for these cases.

Suggestions

• For optimal performance, set aside at least 15 percent (100/8=12.5) of cluster capacity to allow all resources from one physical node to be redistributed to the other seven nodes. This configuration ensures that you have some reserve capacity to perform an upgrade or other AKS day two operations.

• If you want to grow beyond the 200-VM limit for a maximum hardware sized eight-node cluster, increase the size of the AKS host VM. Doubling the size results in roughly double the number of VMs it can manage. In an eight-node Kubernetes cluster, you can manage up to 8,192 (8x1024) VMs based on the recommended resource limits documented in the Maximum supported hardware specifications. You should reserve approximately 30% of capacity, which leaves you with a theoretical limit of 5,734 VMs across all nodes.

  • Standard_D32s_v3, for the AKS host with 32 cores and 128 GB, can support a maximum of 1,600 nodes.

  Note

  Since this configuration isn't extensively tested, it requires a careful approach and validation.

• At a scale like this, you might want to split the environment into at least eight target clusters with 200 worker nodes each.

• To run 200 worker nodes in one target cluster, you can use the default control plane and load balancer size. Depending on the number of pods per node, you can go up at least one size on the control plane and use Standard_D8s_v3.

• Depending on the number of Kubernetes services hosted in each target cluster, you might have to increase the size of the load balancer VM as well at target cluster creation to ensure that services can be reached with high performance and traffic is routed accordingly.

The deployment of AKS on Windows Server distributes the worker nodes for each node pool in a target cluster across the available nodes using placement logic.

If in doubt, contact your local Microsoft office for assistance or post in the community forum.

Next steps

• Storage options