Edit

Share via

How to: Enable secrets encryption for Kubernetes secrets using KMS provider

Applies to: AKS on Azure Local

Kubernetes clusters store small pieces of information called secrets in a persistent Kubernetes secret store (etcd). These secrets often contain sensitive data such as passwords, tokens, or certificates, critical to the operation of applications deployed on K8s. By default, this information is encoded but not encrypted at rest, meaning that if an unauthorized person gains access to the underlying storage, they could potentially decode these secrets. A KMS provider addresses this gap by encrypting the underlying data. Each secret within the secret store is first encrypted with a unique Data Encryption Key (DEK) generated by the K8s API server. This DEK is temporarily held in memory and never written to disk. The API server then requests that the KMS provider encrypt the DEK using a Key Encryption Key (KEK). The final encrypted secret includes both the encrypted data and the encrypted DEK, ensuring that the secrets remain secure and inaccessible without the KEK.

AKS on Azure Local comes with encryption of etcd secrets using a KMS provider. All Kubernetes clusters in Azure Local prior to the 2510 release have a built-in KMS v1 provider enabled by default. This provider generates the Key Encryption Key (KEK) and automatically rotates it every 30 days.

Starting with the 2510 AKS on Azure Local release, KMS v2 is enabled by default on new cluster deployments. Existing clusters continue to use KMS v1. For more information about the KMS v2 provider, see the official guidance.

This article provides information about the KMS provider, verifying encryption and updating secrets after KEK rotation. For more information, see the official Kubernetes documentation for the KMS provider.

KMS v2 on Azure Local 2510 and later

KMS v1 was deprecated with Kubernetes v1.28, and it's recommended that you migrate to KMS v2. The KMS v2 standard eliminates the performance bottlenecks seen in KMS v1 by removing the need for serial decryption during cluster startup, reducing API call overhead and avoiding external key service rate limits. This results in faster cluster initialization and smoother application startup. Other benefits include improved reliability, scalability beyond 2,000 secrets, and enhanced observability through richer health checks.

The KMS Provider v2 is automatically enabled for all new clusters that you create on AKS Arc: there's no action you need to take. The provider generates a new KEK immediately, and then automatically rotates it every 30 days, reducing the risk of key compromise. After each rotation, it uses this new KEK to encrypt the DEK for any subsequent secrets you create. Existing secrets in the cluster aren't automatically re-encrypted. For now, existing clusters continue to use the KMS v1 provider.

Before you begin

Before you begin, ensure that you have the following prerequisites:

Access your Microsoft Entra-enabled cluster

Get the user credentials to access your cluster using the az aksarc get-credentials command. You need the Microsoft.HybridContainerService/provisionedClusterInstances/listUserKubeconfig/action resource, which is included in the Azure Kubernetes Service Arc Cluster User role permission:

az aksarc get-credentials --resource-group $resource_group --name $aks_cluster_name

Verify that the KMS provider is enabled (optional)

This step is optional if you want to verify that the KMS provider is enabled. Run the following command and ensure that the health status of kms-providers is OK:

kubectl get --raw='/readyz?verbose'

[+]ping ok
[+]Log ok
[+]etcd ok
[+]kms-providers ok
[+]poststarthook/start-encryption-provider-config-automatic-reload ok

Verify that the data is encrypted (optional)

This step is optional if you want to verify that secrets and data were encrypted using a KMS provider. See the Kubernetes documentation. You can use the following commands to verify that the data is encrypted:

kubectl exec --stdin --tty <etcd pod name> -n kube-system --etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt get /registry/secrets/default/db-user-pass -w fields

  • kubectl exec: This is the kubectl command used to execute a command inside a running pod. It enables you to run commands within the container of a pod.

  • --stdin: This flag enables you to send input (stdin) to the command you are running inside the pod.

  • --tty: This flag allocates a TTY (terminal) for the command, making it behave as though you're interacting with a terminal session.

  • <etcd pod name>: to find the etcd pod name, run the following command:

    kubectl get pods -n kube-system | findstr etcd-moc

  • -n kube-system: Specifies the namespace where the pod is located. kube-system is the default namespace used by Kubernetes for system components, such as etcd and other control plane services.

  • --etcdctl: Reads the secret from etcd. Additional fields are used for authentication before you get access to etcd.

The following fields are returned in the command output:

"ClusterID" : <cluster id> 
"MemberID" : <member id> 
"Revision" : <revision number> 
"RaftTerm" : 2 
"Key" : <path to the key>
"CreateRevision" : <revision number at the time the key was created> 
"ModRevision" :  <revision number at the time the key was modified> 
"Version" : <version of the key-value pair in etcd> 
"Value" : "k8s:enc:kms:v1:kms-plugin: <encrypted secret value>"  
"Lease" : <lease associated with the secret> 
"More" : <indicates if there are more results> 
"Count" : <number of key-value pairs returned>

After you run the command, examine the Value field in the output in the terminal window. This output shows the value stored in the etcd secret store for this key, which is the encrypted value of the secret. The value is encrypted using a KMS provider. The k8s:enc:kms:v1: prefix indicates that Kubernetes is using the KMS v1 provider to store the secret in an encrypted format.

Note

If you use the kubectl describe secrets command to retrieve secrets, it returns them in base64-encoded format, but unencrypted. The kubectl describe command retrieves the details of a Kubernetes resource via the API server, which manages encryption and decryption automatically. For sensitive data such as secrets, even if they are mounted on a pod, the API server ensures that they are decrypted when accessed. As a result, running the kubectl describe command does not display secrets in their encrypted form, but rather in their decrypted form if they are being used by a resource.

How to update your secrets after KEK is rotated

The KEK is automatically updated every 30 days. At this point, each secret remains encrypted with the KEK that was in use when you created it. When you next update the secret, it's re-encrypted with the current KEK. If you don't regularly update your secret values as part of your regular processes, then consider re-writing them (with the same value) every 30 days anyway. This ensures that you're following Kubernetes best practices and that every secret is encrypted with the latest KEK. For larger clusters, consider updating each namespace's secrets in turn, or developing a script or other automation to streamline the process.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Troubleshooting

If you encounter any errors with the KMS provider, follow the procedure on the Troubleshooting page to troubleshoot the issue.

Next steps

  • Last updated on