Deploy models using Azure CLI and Bicep

• Install the Azure CLI and the cognitiveservices extension for Foundry Tools.

  az extension add -n cognitiveservices
  

• Some commands in this tutorial use the jq tool, which might not be installed on your system. For installation instructions, see Download jq.

• Identify the following information:

  • Your Azure subscription ID

  • Your Foundry Tools resource name

  • The resource group where you deployed the Foundry Tools resource

Add models

To add a model, first identify the model that you want to deploy. Query the available models as follows:

1. Sign in to your Azure subscription.

   az login
   

2. If you have more than one subscription, select the subscription where your resource is located.

   az account set --subscription $subscriptionId
   

3. Set the following environment variables with the name of the Foundry Tools resource you plan to use and resource group.

   accountName="<ai-services-resource-name>"
   resourceGroupName="<resource-group>"
   location="eastus2"
   

4. If you haven't created a Foundry Tools account yet, create one.

   az cognitiveservices account create -n $accountName -g $resourceGroupName --custom-domain $accountName --location $location --kind AIServices --sku S0
   

   Reference: az cognitiveservices account

5. Check which models are available to you and under which SKU. SKUs, also known as deployment types, define how Azure infrastructure processes requests. Models might offer different deployment types. The following command lists all the model definitions available:

   az cognitiveservices account list-models \
       -n $accountName \
       -g $resourceGroupName \
   | jq '.[] | { name: .name, format: .format, version: .version, sku: .skus[0].name, capacity: .skus[0].capacity.default }'
   

   The output includes available models with their properties:

   {
     "name": "Phi-3.5-vision-instruct",
     "format": "Microsoft",
     "version": "2",
     "sku": "GlobalStandard",
     "capacity": 1
   }
   

   Reference: az cognitiveservices account list-models

6. Identify the model you want to deploy. You need the properties name, format, version, and sku. The property format indicates the provider offering the model. Depending on the type of deployment, you might also need capacity.

7. Add the model deployment to the resource. The following example adds Phi-3.5-vision-instruct:

   az cognitiveservices account deployment create \
       -n $accountName \
       -g $resourceGroupName \
       --deployment-name Phi-3.5-vision-instruct \
       --model-name Phi-3.5-vision-instruct \
       --model-version 2 \
       --model-format Microsoft \
       --sku-capacity 1 \
       --sku-name GlobalStandard
   

   Reference: az cognitiveservices account deployment

8. The model is ready to use.

You can deploy the same model multiple times if needed as long as it's under a different deployment name. This capability is useful if you want to test different configurations for a given model, including content filters.

Use the model

You can consume deployed models using the Endpoints for Foundry Models for the resource. When you construct your request, specify the parameter model and insert the model deployment name you created. You can programmatically get the URI for the inference endpoint by using the following code:

Inference endpoint

az cognitiveservices account show  -n $accountName -g $resourceGroupName | jq '.properties.endpoints["Azure AI Model Inference API"]'

To make requests to the Foundry Models endpoint, append the route models. For example: https://<resource>.services.ai.azure.com/models. You can see the API reference for the endpoint at Azure AI Model Inference API reference page.

Inference keys

az cognitiveservices account keys list  -n $accountName -g $resourceGroupName

Manage deployments

You can see all the deployments available using the CLI:

1. Run the following command to see all the active deployments:

   az cognitiveservices account deployment list -n $accountName -g $resourceGroupName
   

   Reference: az cognitiveservices account deployment list

2. You can see the details of a given deployment:

   az cognitiveservices account deployment show \
       --deployment-name "Phi-3.5-vision-instruct" \
       -n $accountName \
       -g $resourceGroupName
   

   Reference: az cognitiveservices account deployment show

3. You can delete a given deployment as follows:

   az cognitiveservices account deployment delete \
       --deployment-name "Phi-3.5-vision-instruct" \
       -n $accountName \
       -g $resourceGroupName
   

   Reference: az cognitiveservices account deployment delete

• Install the Azure CLI.

• Identify the following information:

  • Your Azure subscription ID

• Your Foundry resource (formerly known as Azure AI Services resource) name

• The resource group where the Foundry resource is deployed

• The model name, provider, version, and SKU you want to deploy. You can use the Foundry portal or the Azure CLI to find this information. In this example, you deploy the following model:

  • Model name: Phi-3.5-vision-instruct
  • Provider: Microsoft
  • Version: 2
  • Deployment type: Global standard

Set up the environment

The example in this article is based on code samples contained in the Azure-Samples/azureai-model-inference-bicep repository. To run the commands locally without having to copy or paste file content, clone the repository:

git clone https://github.com/Azure-Samples/azureai-model-inference-bicep

The files for this example are in:

cd azureai-model-inference-bicep/infra

Permissions required to subscribe to Models from Partners and Community

Foundry Models from partners and community available for deployment (for example, Cohere models) require Azure Marketplace. Model providers define the license terms and set the price for use of their models using Azure Marketplace.

When deploying third-party models, ensure you have the following permissions in your account:

Add the model

1. Use the template ai-services-deployment-template.bicep to describe model deployments:

   ai-services-deployment-template.bicep

   @description('Name of the Azure AI services account')
   param accountName string
   
   @description('Name of the model to deploy')
   param modelName string
   
   @description('Version of the model to deploy')
   param modelVersion string
   
   @allowed([
     'AI21 Labs'
     'Cohere'
     'Core42'
     'DeepSeek'
     'xAI'
     'Meta'
     'Microsoft'
     'Mistral AI'
     'OpenAI'
   ])
   @description('Model provider')
   param modelPublisherFormat string
   
   @allowed([
       'GlobalStandard'
       'DataZoneStandard'
       'Standard'
       'GlobalProvisioned'
       'Provisioned'
   ])
   @description('Model deployment SKU name')
   param skuName string = 'GlobalStandard'
   
   @description('Content filter policy name')
   param contentFilterPolicyName string = 'Microsoft.DefaultV2'
   
   @description('Model deployment capacity')
   param capacity int = 1
   
   resource modelDeployment 'Microsoft.CognitiveServices/accounts/deployments@2024-04-01-preview' = {
     name: '${accountName}/${modelName}'
     sku: {
       name: skuName
       capacity: capacity
     }
     properties: {
       model: {
         format: modelPublisherFormat
         name: modelName
         version: modelVersion
       }
       raiPolicyName: contentFilterPolicyName == null ? 'Microsoft.Nill' : contentFilterPolicyName
     }
   }
   

2. Run the deployment:

   RESOURCE_GROUP="<resource-group-name>"
   ACCOUNT_NAME="<azure-ai-model-inference-name>" 
   MODEL_NAME="Phi-3.5-vision-instruct"
   PROVIDER="Microsoft"
   VERSION=2
   
   az deployment group create \
       --resource-group $RESOURCE_GROUP \
       --template-file ai-services-deployment-template.bicep \
       --parameters accountName=$ACCOUNT_NAME modelName=$MODEL_NAME modelVersion=$VERSION modelPublisherFormat=$PROVIDER
   

Use the model

You can consume deployed models using the Endpoints for Foundry Models for the resource. When you construct your request, specify the parameter model and insert the model deployment name you created. You can programmatically get the URI for the inference endpoint by using the following code:

Inference endpoint

az cognitiveservices account show  -n $accountName -g $resourceGroupName | jq '.properties.endpoints["Azure AI Model Inference API"]'

To make requests to the Foundry Models endpoint, append the route models. For example: https://<resource>.services.ai.azure.com/models. You can see the API reference for the endpoint at Azure AI Model Inference API reference page.

Inference keys

az cognitiveservices account keys list  -n $accountName -g $resourceGroupName