Share via

AzureOpenAITokenizerParameters interface

Package:
@azure/search-documents

Properties

allowedSpecialTokens

(Optional) Only applies if the unit is set to azureOpenAITokens. This parameter defines a collection of special tokens that are permitted within the tokenization process.
encoderModelName

Only applies if the unit is set to azureOpenAITokens. Options include 'R50k_base', 'P50k_base', 'P50k_edit' and 'CL100k_base'. The default value is 'CL100k_base'.

Property Details

allowedSpecialTokens

(Optional) Only applies if the unit is set to azureOpenAITokens. This parameter defines a collection of special tokens that are permitted within the tokenization process.

allowedSpecialTokens?: string[]

Property Value

string[]

encoderModelName

Only applies if the unit is set to azureOpenAITokens. Options include 'R50k_base', 'P50k_base', 'P50k_edit' and 'CL100k_base'. The default value is 'CL100k_base'.

encoderModelName?: string

Property Value

string