How to securely deal with secrets in Azure VM Applications

Question

How to securely deal with secrets in Azure VM Applications

Dirk Dulfer 0

I am using VM Applications along with a Policy to ensure VMs have mandatory tools installed.
Some of those tools require API credentials to communicate with the management platform.

Besides scheduled remediation tasks, I would like the VM owners to be able to manually install the VM Application. At their convenience, in their maintenance window. Allowing Reader access to the apps will enable this.

However, API credentials are passed as parameters in the VM Application's install script, which makes them visible to all that have reader (or better) access to that VM app. The documentation does not describe a way how VM Apps can securely store their secrets. I could not discover i.e. Key Vault support.

What is the recommended approach to keep these secrets hidden?

Shweta Mathur 30,426 Reputation points Microsoft Employee Moderator

2023-11-20T04:48:12.01+00:00

Hi @Dirk Dulfer ,

Just checking in to see if the below answer helped. If this answers your query, please don’t forget to click "Accept Answer" which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
Shweta Mathur 30,426 Reputation points Microsoft Employee Moderator

2023-11-28T06:08:58.0233333+00:00

Hi @Dirk Dulfer ,

Just checking in to see if the below answer helped. If this answers your query, please don’t forget to click "Accept Answer" which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

2 answers

Your answer

Shweta Mathur 30,426 Reputation points Microsoft Employee Moderator

2023-11-20T04:48:12.01+00:00

Hi @Dirk Dulfer ,

Just checking in to see if the below answer helped. If this answers your query, please don’t forget to click "Accept Answer" which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
Shweta Mathur 30,426 Reputation points Microsoft Employee Moderator

2023-11-28T06:08:58.0233333+00:00

Hi @Dirk Dulfer ,

Just checking in to see if the below answer helped. If this answers your query, please don’t forget to click "Accept Answer" which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Answer 1

Shweta Mathur 30,426 Microsoft Employee Moderator

Hi @Dirk Dulfer ,

Thanks for reaching out.

You can achieve this is by using Azure Key Vault to store the secrets and then retrieve them during the installation process.

You can use the Custom Script Extension to install the VM application and retrieve the secrets from Key Vault.

You can store sensitive data in a protected configuration, which is encrypted and only decrypted inside the virtual machine. The protected configuration is useful when the execution command includes secrets such as a password or API keys.

Here's an example of how you can use the Custom Script Extension to retrieve secrets from Key Vault:

Create a Key Vault and store the API credentials as a secret in the Key Vault.
Create a managed identity for the VM and grant it access to the Key Vault.
In the Custom Script Extension configuration, specify the script location and the command to be run.
In the protected configuration, specify the Key Vault URL, the secret name, and the managed identity client ID.

Reference - https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows

https://learn.microsoft.com/en-us/azure/key-vault/general/tutorial-net-virtual-machine?tabs=azure-cli

Hope this will help.

Thanks,

Shweta

Please remember to "Accept Answer" if answer helped you.

Dirk Dulfer 0 Reputation points

2023-11-09T12:51:10.5566667+00:00

Thank you Shweta.
I am familiar with this this approach for VM Extensions, which is somewhat cumbersome to execute at scale as all VMs need a managed identity with proper RBAC to the Key Vault to make this work.

One of the requirements is that VM owners/contributors can manually install the VM Application during their preferred maintenance window. VM Apps have proven to be a good and user friendly fit for this, which our users appreciate very much.

In this case however, the script is provided by a third-party vendor and we have a secret to protect. Avoiding having to write and maintain our own installer script, I was hoping the VM Application service had (or will get) a built-in secret management method similar to i.e. Functions or pipeline secrets in DevOps, where the script has placeholders for the secrets. Secrets are read from app version configuration or Key Vault during preparation/execution and obscured in the logs.
Maybe I should raise this as a feature request at Azure Feedback.

Unless you have an alternative suggestion, I will use the guidelines in your answer.
Thanks again.
Shweta Mathur 30,426 Reputation points Microsoft Employee Moderator

2023-11-17T06:57:30.72+00:00

Dirk Dulfer •

Unfortunately, I don't have any alternative as of now as per your requirement.

I would suggest you post this idea at the Azure Feedback Portal, which is monitored by the product team for feature enhancements.

Thank you for your time and patience throughout this issue.

Thanks,

Shweta

Answer 2

Hi @Dirk Dulfer and @Shweta Mathur ,

In case we trust all VM Application and users, then the user assigned managed identity (UAMI) can be used to access Key Vault secrets during the installation.

Since it is possible to configure UAMI so that contributors can assign them to VMs and the managed identity resource has access to Key Vault secrets. So, when a user installs the VM Application, the installation script can assume the managed identity and access the secrets from Key Vault.

The only caveat is that any application running on the VM can assume the managed identity, I think. So, the only protection is the whereabout of the key vault secrets, in the "protected configuration". Hmmm, that´s not much better than passing the secrets in the "protected configuration" directly, I guess.

Still separating the secrets from the installation script is still better, making key rotation easier. Avoiding secrets in version control and install artifacts is a good practice.

The whole answer would be:

Create a Key Vault and store the API credentials as a secret in the Key Vault.
Create a managed identity for the VMs and grant it access to the Key Vault secrets and other resources as needed.
Configure the VM Application / Installation script to
- retrieve the user data (link)
- assume the user assigned managed identity
- retrieve the secrets from Key Vault during installation.
Configure VM´S user data to include information about the key vault secrets to retrieve.
Install the VM Application.

Instead of storing API credentials we could also authorize the managed identity to access the endpoints directly, if the third-party vendor supports that.

What do you think?

Best regards,

Share via

How to securely deal with secrets in Azure VM Applications

2 answers

Your answer