SentencePieceNormalizer Constructor

Definition

Namespace:: Microsoft.ML.Tokenizers

Assembly:: Microsoft.ML.Tokenizers.dll

Package:: Microsoft.ML.Tokenizers v1.0.1

Package:: Microsoft.ML.Tokenizers v0.22.0

Package:: Microsoft.ML.Tokenizers v2.0.0-preview.1.25125.4

Source:: SentencePieceNormalizer.cs

Source:: SentencePieceNormalizer.cs

Source:: SentencePieceNormalizer.cs

Important

Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

Creates a SentencePieceNormalizer object.

public SentencePieceNormalizer(bool removeExtraWhiteSpaces, bool addDummyPrefix, bool escapeWhiteSpaces, bool treatWhitespaceAsSuffix, System.Collections.Generic.IReadOnlyDictionary<string,int>? specialTokens);

new Microsoft.ML.Tokenizers.SentencePieceNormalizer : bool * bool * bool * bool * System.Collections.Generic.IReadOnlyDictionary<string, int> -> Microsoft.ML.Tokenizers.SentencePieceNormalizer

Public Sub New (removeExtraWhiteSpaces As Boolean, addDummyPrefix As Boolean, escapeWhiteSpaces As Boolean, treatWhitespaceAsSuffix As Boolean, specialTokens As IReadOnlyDictionary(Of String, Integer))

Parameters

removeExtraWhiteSpaces: Boolean

addDummyPrefix: Boolean

escapeWhiteSpaces: Boolean

treatWhitespaceAsSuffix: Boolean

specialTokens: IReadOnlyDictionary<String,Int32>

Applies to

Udostępnij przez

SentencePieceNormalizer Constructor

Definition

Parameters

Applies to