Configure container network metrics filtering for Azure Kubernetes Service (AKS) (Preview)

This article shows you how to configure container network metrics filtering for Azure Kubernetes Service (AKS) with Cilium to optimize data collection, reduce storage costs, and focus on the metrics most relevant to your monitoring needs.

Configure container network metrics filtering enables dynamic management of Hubble metrics cardinality through Kubernetes Custom Resource Definitions (CRDs). This feature allows you to dynamically control the cardinality, dimensions, and targets of Hubble metrics without restarting Cilium agents or Prometheus servers.

Prerequisites

• An Azure account with an active subscription. If you don't have one, create a free account before you begin.

• Use the Bash environment in Azure Cloud Shell. For more information, see Get started with Azure Cloud Shell.

  

• If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

  • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Authenticate to Azure using Azure CLI.

  • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use and manage extensions with the Azure CLI.

  • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

• The minimum version of the Azure CLI required to complete the steps in this article is 2.73.0. To find your version, run az --version in the Azure CLI. To install or upgrade, see Install Azure CLI.

• An AKS cluster with Cilium data plane and Advanced Container Networking Services enabled.

• Kubernetes version 1.32 or later.

• Container network metrics filtering works specifically with Cilium data planes.

• The minimum version of the aks-preview Azure CLI extension to complete the steps in this article is 18.0.0b2.

Install the aks-preview Azure CLI extension

Install or update the Azure CLI preview extension by using the az extension add or az extension update command.

# Install the aks-preview extension
az extension add --name aks-preview
# Update the extension to make sure you have the latest version installed
az extension update --name aks-preview

Register the AdvancedNetworkingDynamicMetricsPreview feature flag

1. First, register the AdvancedNetworkingDynamicMetricsPreview feature flag by using the az feature register command:

az feature register --namespace "Microsoft.ContainerService" --name "AdvancedNetworkingDynamicMetricsPreview"

2. Verify successful registration by using the az feature show command. It takes a few minutes for registration to complete.

az feature show --namespace "Microsoft.ContainerService" --name "AdvancedNetworkingDynamicMetricsPreview"

3. When the feature shows Registered, refresh the registration of the Microsoft.ContainerService resource provider by using the az provider register command.

az provider register --namespace Microsoft.ContainerService

Create a new AKS cluster with Cilium

If you already have an existing cluster, you can skip this step.

Use the az aks create command with the --enable-acns flag to create a new AKS cluster that has all Advanced Container Networking Services features. These features include:

• Container Network Observability: Provides insight into your network traffic. To learn more, see Container Network Observability.

• Container Network Security: Offers security features like FQDN filtering. To learn more, see Container Network Security.

# Set environment variables
export RESOURCE_GROUP="cnm-testing-rg"
export CLUSTER_NAME="cnm-cilium-cluster"
export LOCATION="eastus2euap"  # Use canary region for preview features

# Create resource group
az group create --name $RESOURCE_GROUP --location $LOCATION

# Create AKS cluster with Cilium and ACNS enabled
az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $CLUSTER_NAME \
    --location $LOCATION \
    --network-plugin azure \
    --network-dataplane cilium \
    --enable-acns \
    --enable-managed-identity \
    --generate-ssh-keys \
    --kubernetes-version 1.32

Get cluster credentials

Get your cluster credentials by using the az aks get-credentials command:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --overwrite-existing

Configure custom resources for metrics filtering

Container network metrics filtering uses the ContainerNetworkMetric Custom Resource Definition (CRD) to define filtering rules. Only one CRD can exist per cluster, and changes take approximately 30 seconds to reconcile. If the CRD is not applied, all metrics will be collected.

apiVersion: acn.azure.com/v1alpha1
kind: ContainerNetworkMetric
metadata:
  name: : container-network-metric # Cluster scoped
spec:
  filters:
  - metric: flow
    includeFilters: # List of include filters
      - name: sample-filter # Filter name
        from:
          namespacedPod: # List of source namespace/pods. Prepend namespace with /
            - sample-namespace/sample-pod
          labelSelector: # Standard k8s label selector
            matchLabels:
              app: frontend
              k8s.io/namespace: sample-namespace
            matchExpressions:
              - key: environment
                operator: In
                values:
                  - production
                  - staging
          ip: # List of source IPs; can be CIDR
            - "192.168.1.10"
            - "10.0.0.1"
        to:
          namespacedPod:
            - sample-namespace2/sample-pod2
          labelSelector:
            matchLabels:
              app: backend
              k8s.io/namespace: sample-namespace2
            matchExpressions:
              - key: tier
                operator: NotIn
                values:
                  - dev
          ip:
            - "192.168.1.20"
            - "10.0.1.1"
        protocol: # List of protocols; can be tcp, udp, dns
          - tcp
          - udp
          - dns
        verdict: # List of verdicts; can be forwarded, dropped
          - forwarded
          - dropped
    metric: dns
    excludeFilters: # List of exclude filters
      - name: sample-filter # Filter name
        from:
          namespacedPod: # List of source namespace/pods. Prepend namespace with /
            - sample-namespace/sample-pod
          labelSelector: # Standard k8s label selector
            matchLabels:
              app: frontend
              k8s.io/namespace: sample-namespace

The following table describes the fields in the custom resource definition:

Apply the ContainerNetworkMetric custom resource to enable log collection at the cluster:

kubectl apply -f <crd.yaml>

Clean up and reset

To clean up filtering configuration:

# Delete the CRD
kubectl delete ContainerNetworkMetric container-network-metric

Example filtering configuration

1. Create a basic filtering configuration that focuses on DNS metrics:

# basic-dns-filter.yaml
apiVersion: acn.azure.com/v1alpha1
kind: ContainerNetworkMetric
metadata:
  name: container-network-metric
spec:
  filters:
    - metric: dns  # Supported: dns, flow, tcp, drop
      excludeFilters:
        - from:
            namespacedPod: ["kube-system/coredns"]

2. Configure filtering for TCP metrics with include and exclude filters:

# tcp-metrics-filter.yaml
apiVersion: acn.azure.com/v1alpha1
kind: ContainerNetworkMetric
metadata:
  name: container-network-metric
spec:
  filters:
    - metric: tcp
      includeFilters:
          - from:
            labelSelector:
              matchLabels:
                tier: "frontend"
      excludeFilters:
        - to:
            namespacedPod: ["kube-system/metrics-server"]

3. Configure filtering for network flow metrics:

# flow-metrics-filter.yaml
apiVersion: acn.azure.com/v1alpha1
kind: ContainerNetworkMetric
metadata:
  name: container-network-metric
spec:
  filters:
    - metric: flow
      includeFilters:
        - from:
            labelSelector:
              matchLabels:
                tier: "frontend"
        - to:
            labelSelector:
              matchLabels:
                tier: "backend"
      excludeFilters:
        - from:
            namespacedPod: ["default/test"]

4. Configure filtering for dropped packet metrics:

# drop-metrics-filter.yaml
apiVersion: acn.azure.com/v1alpha1
kind: ContainerNetworkMetric
metadata:
  name: container-network-metric
spec:
  filters:
    - metric: drop
      excludeFilters:
        - from:
            namespacedPod: ["kube-system/"]

5. Configure filtering for multiple metric types in a single CRD:

# multi-metrics-filter.yaml
apiVersion: acn.azure.com/v1alpha1
kind: ContainerNetworkMetric
metadata:
  name: container-network-metric
spec:
  filters:
    - metric: dns
      includeFilters:
        - protocol: ["dns"]
      excludeFilters:
        - from:
            namespacedPod: ["kube-system/*"]
    - metric: tcp
      includeFilters:
        - protocol: ["tcp"]
        - from:
            labelSelector:
              matchLabels:
                environment: "production"
    - metric: flow
      excludeFilters:
        - from:
            namespacedPod: ["default/debug-*"]
    - metric: drop
      includeFilters:
        - reason: ["Policy denied", "Invalid"]

Best practices

• Ensure you do not have conflicting include and exclude filter on the CRD.

• Leverage Kubernetes label selectors for flexible filtering.

• Always validate filtering configurations in development or staging

• Regularly review filtered metrics to ensure important data isn't lost.

• Remember that only one CRD can exist per cluster.

Troubleshooting

Common issues

Issue: CRD configuration not applied

Solution: Check that the feature flag is registered and only one CRD exists:

# Verify feature registration
az feature show --namespace "Microsoft.ContainerService" --name "AdvancedNetworkingDynamicMetricsPreview"

# Check existing CRDs
kubectl get ContainerNetworkMetric

Issue: Metrics still showing after applying excludeFilters

Solution: Remember that preexisting metrics persist in Prometheus. You may need to wait for new metrics to be generated to see the filtering effects.

Limitations

• This feature is specifically designed for Cilium data planes only
• Only one ContainerNetworkMetric CRD can exist per cluster
• Preexisting metrics persist in Prometheus; new filtering rules apply to newly generated metrics
• Requires Kubernetes version 1.32 or later

Related content

• To learn about container network metrics capabilities, see Container network metrics overview.
• To create an AKS cluster with Advanced Container Networking Services, see Set up Container Network Observability for AKS.
• Get more information about Advanced Container Networking Services for AKS.